Command line

The promptfoo command line utility includes these command groups:

• init [directory] - Initialize a new project with prompts, providers, and test cases.
• eval - Evaluate prompts and models. This is the command you'll be using the most!
  • eval setup
• view - Start a browser UI for visualization of results.
• share [id] - Create a URL that can be shared online for an eval or model audit.
• auth - Manage authentication for cloud features.
  • auth login
  • auth logout
  • auth whoami
  • auth can-create-targets
  • auth teams
• cache - Manage cache.
  • cache clear
• code-scans - Scan code changes for LLM security vulnerabilities.
  • code-scans run
• config - Edit configuration settings.
  • config get
  • config set
  • config unset
• debug - Display debug information for troubleshooting.
• generate - Generate data.
  • generate dataset
  • generate redteam
  • generate assertions
• list - List various resources like evals, prompts, and datasets.
  • list evals
  • list prompts
  • list datasets
• logs - View promptfoo log files.
  • logs [file]
  • logs list
• mcp - Start a Model Context Protocol (MCP) server to expose promptfoo tools to AI agents and development environments.
• scan-model - Scan ML models for security vulnerabilities.
• show [id] - Show details of a specific resource (eval, prompt, or dataset).
• delete <id> - Delete an eval by ID; accepts latest or all.
• retry <evalId> - Retry ERROR results from a previous eval in place.
• validate - Validate a promptfoo configuration file.
  • validate config
  • validate target
• feedback <message> - Send feedback to the Promptfoo developers.
• import <filepath> - Import an eval file from JSON format.
• export - Export eval records or logs.
  • export eval <evalId>
  • export logs
• redteam - Red team LLM applications.
  • redteam init
  • redteam setup
  • redteam run
  • redteam discover
  • redteam generate
  • redteam poison
  • redteam eval
  • redteam report
  • redteam plugins

Common Options

Most commands support the following common options:

:::note

For promptfoo eval, -v is the short form of --vars, not --verbose. Use the long form (--verbose) or set LOG_LEVEL=debug to enable debug logs during eval runs.

:::

Multiple Environment Files

You can load multiple .env files. Later files override values from earlier ones:

# Repeated flags
promptfoo eval --env-file .env --env-file .env.local

# Comma-separated
promptfoo eval --env-file .env,.env.local

All specified files must exist or an error is thrown.

promptfoo eval

By default the eval command will read the promptfooconfig.yaml configuration file in your current directory. But, if you're looking to override certain parameters you can supply optional arguments:

The eval command will return exit code 100 when there is at least 1 test case failure or when the pass rate is below the threshold set by PROMPTFOO_PASS_RATE_THRESHOLD. It will return exit code 1 for any other error. The exit code for failed tests can be overridden with environment variable PROMPTFOO_FAILED_TEST_EXIT_CODE.

Pause and Resume

promptfoo eval --resume            # resumes the latest eval
promptfoo eval --resume <evalId>   # resumes a specific eval

• On resume, promptfoo reuses the original run's effective runtime options (e.g., --delay, --no-cache, --max-concurrency, --repeat), skips completed test/prompt pairs, ignores CLI flags that change test ordering to keep indices aligned, and disables watch mode.

Retry Errors

promptfoo eval --retry-errors      # retries all ERROR results from the latest eval

• The retry errors feature automatically finds ERROR results from the latest eval and re-runs only those test cases. This is useful when evals fail due to temporary network issues, rate limits, or API errors.
• Data safety: If the retry fails, your original ERROR results are preserved. Old ERROR results are only removed after the retry succeeds. You can safely run --retry-errors again if it fails.
• Cannot be used together with --resume or --no-write flags.
• Uses the original eval's configuration and runtime options to ensure consistency.

promptfoo init [directory]

Set up a new promptfoo project with prompts, providers, and test cases.

promptfoo view

Start a browser UI for visualization of results.

If you've used PROMPTFOO_CONFIG_DIR to override the promptfoo output directory, run promptfoo view [directory].

promptfoo share [id] {#promptfoo-share-evalid}

Create a URL that can be shared online. If no ID is provided, promptfoo shares the most recent eval or model audit.

promptfoo cache

Manage cache.

promptfoo feedback <message>

Send feedback to the promptfoo developers.

promptfoo list

List various resources like evals, prompts, and datasets.

promptfoo logs

View promptfoo log files directly from the command line.

promptfoo logs list

List available log files in the logs directory.

Examples

# View most recent log file (or current session's log if run during a CLI session)
promptfoo logs

# View last 50 lines
promptfoo logs -n 50

# View first 20 lines
promptfoo logs --head 20

# Follow log in real-time (like tail -f)
promptfoo logs -f

# List all available log files
promptfoo logs --list

# View a specific log file by name or partial match
promptfoo logs promptfoo-debug-2024-01-15_10-30-00.log
promptfoo logs 2024-01-15

# View error logs only
promptfoo logs --type error

# Filter logs by pattern (case-insensitive regex)
promptfoo logs --grep "error|warn"
promptfoo logs --grep "openai"

Log files are stored in ~/.promptfoo/logs by default. Set PROMPTFOO_LOG_DIR to use a custom directory.

promptfoo code-scans run [repo-path]

Scan code changes for LLM security vulnerabilities.

For complete setup and JSON output details, see the code scanning CLI docs.

promptfoo mcp

Start a Model Context Protocol (MCP) server to expose promptfoo's eval and testing capabilities as tools that AI agents and development environments can use.

Transport Types

• STDIO: Best for desktop AI tools like Cursor, Claude Desktop, and local AI agents that communicate via standard input/output
• HTTP: Best for web applications, APIs, and remote integrations that need HTTP endpoints

Examples

# Start MCP server with STDIO transport (for Cursor, Claude Desktop, etc.)
npx promptfoo@latest mcp --transport stdio

# Start MCP server with HTTP transport on default port
npx promptfoo@latest mcp --transport http

# Start MCP server with HTTP transport on custom port
npx promptfoo@latest mcp --transport http --port 8080

Available Tools

The MCP server provides 14 tools for AI agents:

Core Eval Tools:

• list_evaluations - Browse your eval runs with optional dataset filtering
• get_evaluation_details - Get comprehensive results, metrics, and test cases for a specific eval
• run_evaluation - Run evals with custom parameters, test case filtering, and concurrency control
• share_evaluation - Generate publicly shareable URLs for eval results

Redteam Security Tools:

• redteam_run - Execute comprehensive security testing against AI applications with dynamic attack probes
• redteam_generate - Generate adversarial test cases for redteam security testing with configurable plugins and strategies

Configuration & Testing:

• validate_promptfoo_config - Validate configuration files using the same logic as the CLI
• test_provider - Test AI provider connectivity, credentials, and response quality
• run_assertion - Test individual assertion rules against outputs for debugging

Generation Tools:

• generate_dataset - Generate test datasets using AI
• generate_test_cases - Generate test cases with assertions for existing prompts
• compare_providers - Compare multiple AI providers side-by-side

Debugging Tools:

• list_logs - List promptfoo log files with metadata
• read_logs - Read promptfoo log file contents with filtering options

For detailed setup instructions and integration examples, see the MCP Server documentation.

promptfoo show <id>

Show details of a specific resource. If no ID is provided, promptfoo shows the most recent eval.

promptfoo delete <id>

Deletes a specific resource.

promptfoo retry <evalId>

Retry all ERROR results from a specific eval. This command finds test cases that resulted in errors (e.g., from network issues, rate limits, or API failures) and re-runs only those test cases. The results are updated in place in the original eval.

Examples:

# Retry errors from a specific eval
promptfoo retry eval-abc123

# Retry with a different config file
promptfoo retry eval-abc123 -c updated-config.yaml

# Retry with verbose output and limited concurrency
promptfoo retry eval-abc123 -v --max-concurrency 2

# Retry and share results to cloud
promptfoo retry eval-abc123 --share

:::tip Data Safety If the retry operation fails (network error, API timeout, etc.), your original ERROR results are preserved. You can simply run the retry command again to continue. Old ERROR results are only removed after the retry succeeds. :::

:::tip Unlike --filter-errors-only which creates a new eval, promptfoo retry updates the original eval in place. Use retry when you want to fix errors in an existing eval without creating duplicates. :::

promptfoo import <filepath>

Import an eval file from JSON format.

When importing an eval, the following data is preserved from the export:

• Eval ID - Preserved by default. Use --new-id to generate a new ID, or --force to replace an existing eval.
• Timestamp - The original creation timestamp is always preserved (even with --new-id or --force)
• Author - The original author is always preserved (even with --new-id or --force)
• Config, results, and all test data - Fully preserved

If an eval with the same ID already exists, the import will fail with an error unless you specify --new-id (to create a duplicate with a new ID) or --force (to replace the existing eval).

Example:

# Import an eval, preserving the original ID
promptfoo import my-eval.json

# Import even if an eval with this ID exists (creates duplicate with new ID)
promptfoo import --new-id my-eval.json

# Replace an existing eval with updated data
promptfoo import --force my-eval.json

promptfoo export

Export eval records or logs.

promptfoo export eval <evalId>

Export an eval record to JSON format. To export the most recent, use latest.

promptfoo export logs

Collect and zip log files for debugging purposes.

This command creates a compressed tar.gz archive containing your promptfoo log files, making it easy to share them for debugging purposes. If no output path is specified, it will generate a timestamped filename automatically.

Log files are stored in ~/.promptfoo/logs by default. To use a custom log directory, set the PROMPTFOO_LOG_DIR environment variable.

promptfoo validate

Validate a promptfoo configuration file to ensure it follows the correct schema and structure.

promptfoo validate config

This command validates both the configuration file and the test suite to ensure they conform to the expected schema. It will report any validation errors with detailed messages to help you fix configuration issues.

Examples:

# Validate the default promptfooconfig.yaml
promptfoo validate

# Validate a specific configuration file
promptfoo validate -c my-config.yaml

# Validate multiple configuration files
promptfoo validate -c config1.yaml config2.yaml

The command will exit with code 1 if validation fails, making it useful for CI/CD pipelines to catch configuration errors early.

promptfoo validate target

Examples:

# Test all providers in a config
promptfoo validate target -c promptfooconfig.yaml

# Test one provider by ID or cloud UUID
promptfoo validate target -t openai:gpt-5-mini

promptfoo scan-model

Scan ML models for security vulnerabilities. Provide one or more paths to model files or directories.

By default, scan-model saves model audit records to the promptfoo database. In that mode, --output writes promptfoo's JSON result payload even when --format sarif is selected. Use --no-write --format sarif --output results.sarif for raw ModelAudit SARIF output.

For model source examples, scanner IDs, SBOM output, and SARIF workflows, see the ModelAudit docs.

promptfoo auth

Manage authentication for cloud features.

promptfoo auth login

Login to the promptfoo cloud.

After login, if you have multiple teams, you can switch between them using the teams subcommand.

promptfoo auth logout

Logout from the promptfoo cloud.

promptfoo auth whoami

Display current authentication status including user, organization, and active team.

Output includes:

• User email
• Organization name
• Current team (if logged in to a multi-team organization)
• App URL

Example:

promptfoo auth whoami

Output:

Currently logged in as:
User: [email protected]
Organization: Acme Corp
Current Team: Engineering Team
App URL: https://www.promptfoo.app

promptfoo auth can-create-targets

Check whether the current user can create cloud targets.

promptfoo auth teams

Manage team switching for organizations with multiple teams.

promptfoo auth teams list

List all teams you have access to in the current organization.

promptfoo auth teams current

Show the currently active team.

promptfoo auth teams set <teamIdentifier>

Switch to a specific team. The team identifier can be:

• Team name (e.g., "Engineering")
• Team slug (e.g., "engineering")
• Team ID (e.g., "team_12345")

Examples:

# Switch to team by name
promptfoo auth teams set "Engineering Team"

# Switch to team by slug
promptfoo auth teams set engineering

# Switch to team by ID
promptfoo auth teams set team_12345

Your team selection is remembered across CLI sessions and applies to all promptfoo operations including evals and red team testing.

Team Selection Across Organizations

If you have access to multiple organizations, team selections are isolated per organization. This means:

• Each organization remembers its own team selection
• Switching between organizations preserves your team choice in each org
• When you log into an organization, your previously selected team is automatically restored

Example workflow:

# Login to Organization A
promptfoo auth login --api-key <org-a-key>
promptfoo auth teams set "Engineering"     # Set team in Org A

# Login to Organization B
promptfoo auth login --api-key <org-b-key>
promptfoo auth teams set "Marketing"       # Set team in Org B

# Login back to Organization A
promptfoo auth login --api-key <org-a-key>
promptfoo auth teams current              # Shows "Engineering" (preserved!)

Your team selection persists across login sessions within the same organization.

promptfoo config

Edit configuration settings.

promptfoo config get email

Get the user's email address.

promptfoo config set email <email>

Set the user's email address.

promptfoo config unset email

Unset the user's email address.

promptfoo debug

Display debug information for troubleshooting.

promptfoo generate dataset

BETA: Generate synthetic test cases based on existing prompts and variables.

For example, this command will modify your default config file (usually promptfooconfig.yaml) with new test cases:

promptfoo generate dataset -w

This command will generate test cases for a specific config and write them to a file, while following special instructions:

promptfoo generate dataset -c my_config.yaml -o new_tests.yaml -i 'All test cases for {{location}} must be European cities'

promptfoo generate assertions

Generate additional objective/subjective assertions based on existing prompts and assertions.

• This command can be used to generate initial set of assertions, if none exist.
• Will only add non-overlapping, independent assertions
• Generates both python and natural language assertions.

When brainstorming assertions:

• Generates python code for any objective assertions
• Uses a specified natural language assertion type (pi, llm-rubric, or g-eval) for any subjective assertion.

For example, this command will modify your default config file (usually promptfooconfig.yaml) with new test cases:

promptfoo generate assertions -w

This command will generate pi and python assertions for a specific config and write them to a file, while following special instructions:

promptfoo generate assertions -c my_config.yaml -o new_tests.yaml -i 'I need assertions about pronunciation'

promptfoo generate redteam

Alias for promptfoo redteam generate.

promptfoo redteam init

Initialize a red teaming project.

Example:

promptfoo redteam init my_project

:::danger Adversarial testing produces offensive, toxic, and harmful test inputs, and may cause your system to produce harmful outputs. :::

For more detail, see red team configuration.

promptfoo redteam setup

Start browser UI and open to red team setup.

promptfoo redteam run

Run the complete red teaming process (init, generate, and evaluate).

promptfoo redteam discover

Runs the Target Discovery Agent against your application.

:::info

Only a configuration file or target can be specified

:::

promptfoo redteam generate

Generate adversarial test cases to challenge your prompts and models.

For example, let's suppose we have the following promptfooconfig.yaml:

prompts:
  - 'Act as a trip planner and help the user plan their trip'

providers:
  - openai:gpt-5-mini
  - openai:gpt-5

This command will generate adversarial test cases and write them to redteam.yaml.

promptfoo redteam generate

This command overrides the system purpose and the variable to inject adversarial user input:

promptfoo redteam generate --purpose 'Travel agent that helps users plan trips' --injectVar 'message'

promptfoo redteam poison

Generate poisoned documents for RAG testing.

promptfoo redteam eval

Works the same as promptfoo eval, but defaults to loading redteam.yaml.

promptfoo redteam report

Start a browser UI and open the red teaming report.

Example:

promptfoo redteam report -p 8080

promptfoo redteam plugins

List all available red team plugins.

Specifying Command Line Options in Config

Many command line options can be specified directly in your promptfooconfig.yaml file using the commandLineOptions section. This is convenient for options you frequently use or want to set as defaults for your project.

Example:

prompts:
  - Write a funny tweet about {{topic}}
providers:
  - openai:o3-mini
tests:
  - file://test_cases.csv

# Command line options as defaults
commandLineOptions:
  maxConcurrency: 5
  repeat: 2
  delay: 250
  share: false
  cache: true
  write: true

With this configuration, you can simply run promptfoo eval without specifying these options on the command line. You can still override these settings by providing the corresponding flag when running the command.

ASCII-only outputs

To disable terminal colors for printed outputs, set FORCE_COLOR=0 (this is supported by the chalk library).

For the eval command, you may also want to disable the progress bar and table as well, because they use special characters:

FORCE_COLOR=0 promptfoo eval --no-progress-bar --no-table

Environment variables

These general-purpose environment variables are supported:

:::tip promptfoo will load environment variables from the .env in your current working directory. :::

:::tip For detailed information on using timeout features, including configuration examples and troubleshooting tips, see Timeout errors in the troubleshooting guide. :::