site/docs/faq.md
Promptfoo is a local-first, open-source tool designed to help evaluate (eval) large language models (LLMs). Promptfoo is designed for application developers and for business applications. It features a simple, flexible, and extensible API. With Promptfoo you can:
LLM red teaming is the process of systematically testing LLMs to identify potential vulnerabilities, weaknesses, and unintended behaviors before deployment. Promptfoo supports this by offering a framework for generating and executing adversarial tests, aligned with industry standards like OWASP LLM Top 10 and NIST AI Risk Management Framework.
Promptfoo's red teaming capabilities allow you to:
For more details, see our LLM Red Teaming Guide.
Promptfoo supports a wide range of LLM providers, including:
Promptfoo's flexible architecture allows for easy integration with new or custom LLM providers. For the most up-to-date list and integration instructions, please refer to our Providers documentation.
By default, no — the open-source eval runner runs on your machine, and configured provider calls are sent directly to that provider. Promptfoo-operated services may receive data when you opt into a Cloud-backed feature, including hosted generation or grading, red team target/provider setup helpers, red team target/provider test requests, sharing, Cloud sync, hosted reports, telemetry, and account/license checks.
By default, no — API keys are read from local environment variables and are not transmitted on their own. If you paste credentials directly into configs, request examples, red team target/provider setup forms, or shareable eval data, those values are treated like any other config field and may be saved in local artifacts or sent to configured providers, sharing endpoints, or Promptfoo-hosted features that process that data.
By default, eval results are stored locally. Inputs and outputs may leave your machine when you use hosted generation or grading, sharing, Cloud sync, hosted reports, red team target/provider setup helpers, red team target/provider test requests, or other Cloud-backed features.
We do not collect PII through product analytics. The email or user identifier associated with your local Promptfoo environment may be sent to the relevant Promptfoo-operated endpoint when you sign in to Promptfoo Cloud, share evals, invoke a Cloud-backed feature, or set PROMPTFOO_DISABLE_TELEMETRY (which records a one-time opt-out acknowledgment so opt-out usage can be measured). See the security policy for the full list of Cloud-backed features and the data each one sends.
Promptfoo proxy settings are configured through environment variables:
HTTP_PROXY: For HTTP requestsHTTPS_PROXY: For HTTPS requestsNO_PROXY: Comma-separated list of hosts to exclude from proxyingThe proxy URL format is: [protocol://][user:password@]host[:port]
For example:
# Basic proxy
export HTTPS_PROXY=http://proxy.company.com:8080
# Proxy with authentication
export HTTPS_PROXY=http://username:[email protected]:8080
# Exclude specific hosts from proxying
export NO_PROXY=localhost,127.0.0.1,internal.domain.com
Note: Environment variables are specific to your terminal/shell instance. If you need them permanently, add them to your shell's startup file (e.g., ~/.bashrc, ~/.zshrc).
For environments with custom certificate authorities (like corporate environments), configure SSL/TLS settings using these environment variables:
PROMPTFOO_CA_CERT_PATH: Path to a custom CA certificate bundle. The path can be absolute or relative to your working directory. Invalid paths will log a warning:
# Absolute path
export PROMPTFOO_CA_CERT_PATH=/path/to/ca-bundle.crt
# Relative path
export PROMPTFOO_CA_CERT_PATH=./certs/ca-bundle.crt
PROMPTFOO_INSECURE_SSL: Set to true to disable SSL certificate verification:
export PROMPTFOO_INSECURE_SSL=true
Remember that like all environment variables, these settings are specific to your terminal/shell instance.
Promptfoo can be integrated into CI/CD pipelines via GitHub Action, used with testing frameworks like Jest and Vitest, and incorporated into various stages of the development process.
Set the following environment variables before running the CLI to turn off common hosted features:
export PROMPTFOO_DISABLE_TELEMETRY=1
export PROMPTFOO_DISABLE_UPDATE=1
export PROMPTFOO_DISABLE_REMOTE_GENERATION=true
export PROMPTFOO_DISABLE_SHARING=1
export PROMPTFOO_SELF_HOSTED=1
Only configure local or self-hosted LLM providers (e.g., Ollama) so the CLI does not attempt to reach external APIs.
These settings are not a network firewall. For strict offline or air-gapped use, combine local/self-hosted providers with network egress controls or use Promptfoo Enterprise On-Prem.
Yes. The documentation website follows the LLMs.txt specification so automated tools can easily index our content. You can access the files at:
Usage with AI assistants: Copy the llms-full.txt content into your AI assistant (ChatGPT, Claude, etc.) for comprehensive promptfoo context when working on LLM evaluations, red-teaming, or configuration questions.