OpenAI

To use the OpenAI API, set the OPENAI_API_KEY environment variable, specify the apiKey field in the configuration file, or pass the API key as an argument to the constructor.

Example:

export OPENAI_API_KEY=your_api_key_here

The OpenAI provider supports the following model formats:

• openai:<model name> - auto-routes known OpenAI model IDs to their supported promptfoo provider (chat, completions, speech, realtime, or responses); unknown model names default to Chat Completions. Use an explicit endpoint prefix when you need deterministic routing.
• openai:chat:<model name> - uses chat models against the /v1/chat/completions endpoint
• openai:responses:<model name> - uses responses API models over HTTP connections
• openai:assistant:<assistant id> - use an assistant
• openai:chat - defaults to gpt-4.1-2025-04-14
• openai:responses - defaults to gpt-4.1-2025-04-14
• openai:chat:ft:gpt-4.1-mini-2025-04-14:company-name::ID - example of a fine-tuned chat completion model
• openai:completion - defaults to gpt-3.5-turbo-instruct
• openai:completion:<model name> - uses any model name against the /v1/completions endpoint
• openai:embedding:<model name> / openai:embeddings:<model name> - uses any model name against the /v1/embeddings endpoint
• openai:moderation:<model name> - uses moderation models (default: omni-moderation-latest)
• openai:image:<model name> - uses image generation models
• openai:transcription:<model name> - uses audio transcription models
• openai:tts / openai:speech - defaults to gpt-4o-mini-tts on the audio speech endpoint
• openai:tts:<model name> / openai:speech:<model name> - uses text-to-speech models
• openai:realtime - defaults to gpt-realtime-1.5
• openai:realtime:<model name> - uses realtime API models over WebSocket connections
• openai:video:<model name> - uses Sora video generation models
• openai:agents:<agent name> - runs agentic workflows via OpenAI Agents SDK
• openai:chatkit:<workflow_id> - runs ChatKit workflows
• openai:codex-sdk / openai:codex - runs agentic coding workflows via OpenAI Codex SDK, with optional inline model selection like openai:codex:gpt-5.5
• openai:codex-app-server / openai:codex-desktop - runs the experimental Codex app-server protocol for rich-client event, approval, sandbox, skill, plugin, and thread lifecycle evals

The openai:<endpoint>:<model name> construction is useful for newly released or custom models. Specify the endpoint explicitly when the model is not yet recognized by promptfoo's auto-routing.

providers:
  - id: openai:chat:gpt-5
    config:
      verbosity: high # low | medium | high
      reasoning_effort: minimal # GPT-5.5 uses none instead
  # For the Responses API, use a nested reasoning object:
  - id: openai:responses:gpt-5
    config:
      reasoning:
        effort: minimal # GPT-5.5 uses none instead

The OpenAI provider supports a handful of configuration options, such as temperature, max_tokens, max_completion_tokens, functions, and tools, which can be used to customize model behavior like so:

providers:
  - id: openai:chat:gpt-4.1-mini
    config:
      temperature: 0
      max_tokens: 1024
  - id: openai:chat:gpt-5.5
    config:
      max_completion_tokens: 1024

Note: OpenAI models can also be accessed through Azure OpenAI, which offers additional enterprise features, compliance options, and regional availability.

Requests sent by built-in OpenAI providers to the OpenAI API include the X-OpenAI-Originator: promptfoo header for source attribution. To route requests with a different originator value, override this header through the headers configuration option.

Formatting chat messages

For information on setting up chat conversation, see chat threads.

Configuring parameters

The providers list takes a config key that allows you to set parameters like temperature for non-reasoning models, max_tokens, max_completion_tokens for GPT-5 family chat models, and others. For example:

providers:
  - id: openai:chat:gpt-4.1-mini
    config:
      temperature: 0
      max_tokens: 128
  - id: openai:chat:gpt-5.5
    config:
      max_completion_tokens: 128
      apiKey: '{{ env.OPENAI_API_KEY }}'

Supported parameters include:

Use inputCost and outputCost when a model has different prompt and completion rates. The legacy cost option remains a shared fallback. For audio-capable models, audioInputCost and audioOutputCost take precedence over audioCost.

Here are the type declarations of config parameters:

interface OpenAiConfig {
  // Completion parameters
  temperature?: number;
  max_tokens?: number;
  max_completion_tokens?: number;
  reasoning?: {
    effort?: 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' | 'max' | null;
    context?: 'auto' | 'current_turn' | 'all_turns' | null;
    mode?: 'standard' | 'pro' | null;
    summary?: 'auto' | 'concise' | 'detailed' | null;
  };
  top_p?: number;
  frequency_penalty?: number;
  presence_penalty?: number;
  best_of?: number;
  functions?: OpenAiFunction[];
  function_call?: 'none' | 'auto' | { name: string };
  tools?: OpenAiTool[];
  tool_choice?: 'none' | 'auto' | 'required' | { type: 'function'; function?: { name: string } };
  response_format?: { type: 'json_object' | 'json_schema'; json_schema?: object };
  stop?: string[];
  seed?: number;
  user?: string;
  metadata?: Record<string, string>;
  store?: boolean;
  prompt_cache_key?: string;
  prompt_cache_options?: {
    mode?: 'implicit' | 'explicit';
    ttl?: '30m';
  };
  prompt_cache_retention?: 'in_memory' | '24h' | null;
  passthrough?: object;

  // Function tool callbacks
  functionToolCallbacks?: Record<
    OpenAI.FunctionDefinition['name'],
    (arg: string) => Promise<string>
  >;

  // General OpenAI parameters
  apiKey?: string;
  apiKeyEnvar?: string;
  apiHost?: string;
  apiBaseUrl?: string;
  organization?: string;
  cost?: number;
  inputCost?: number;
  outputCost?: number;
  audioCost?: number;
  audioInputCost?: number;
  audioOutputCost?: number;
  headers?: { [key: string]: string };
  maxRetries?: number;
}

Generating Multiple Responses

Use passthrough to set OpenAI's n parameter for generating multiple responses in a single request:

providers:
  - id: openai:chat:gpt-4o
    config:
      passthrough:
        n: 3 # Generate 3 responses

When n > 1, the primary output contains the first choice's content, and all generated choices are available in the response metadata under metadata.choices. Each choice includes the full response object with message, finish_reason, and index.

Reducing Embedding Dimensions

Use passthrough to send raw Embeddings API fields such as dimensions. OpenAI supports dimensions on text-embedding-3 and later models when you want a smaller vector size:

providers:
  - id: openai:embedding:text-embedding-3-large
    config:
      passthrough:
        dimensions: 1024

Models

OpenAI updates aliases, dated snapshots, and pricing frequently. Promptfoo supports explicit endpoint syntax like openai:chat:<model> and openai:responses:<model> for newly released models right away, while the tables below call out the common model IDs promptfoo knows about for routing and cost estimation. Check the official OpenAI models docs and API pricing for the latest availability and rates.

Fine-tuned and legacy completion models

Fine-tuned model IDs can be used with their corresponding endpoint, for example openai:chat:ft:gpt-4.1-mini-2025-04-14:company-name::ID or openai:completion:ft:babbage-002:company-name::ID. Promptfoo recognizes the published inference rates for fine-tuned GPT-4.1, GPT-4o, GPT-3.5, GPT-4, o4-mini, Babbage, and Davinci bases, including cached-input and Batch rates where available. Flex and Priority pricing is not inferred for fine-tuned models. See the OpenAI supervised fine-tuning guide for currently supported training bases and lifecycle information.

The empty segment before ID represents an omitted fine-tuning suffix. If a suffix was set during training, use ft:<base-model>:<organization>:<suffix>:<id> instead.

The legacy /v1/completions endpoint supports babbage-002, davinci-002, gpt-3.5-turbo-instruct, and gpt-3.5-turbo-instruct-0914. These models are useful for maintaining existing evals, but OpenAI has scheduled the legacy snapshots for removal on September 28, 2026; prefer a current Chat Completions or Responses model for new work.

GPT-4.1

GPT-4.1 is a capable long-context model for complex tasks with a 1,047,576 token context window and 32,768 max output tokens. It is available in three variants with different price points:

All variants support text and image input with text output and have a May 31, 2024 knowledge cutoff.

Usage Examples

Standard model:

providers:
  - id: openai:chat:gpt-4.1 # or openai:responses:gpt-4.1
    config:
      temperature: 0.7

More affordable variants:

providers:
  - id: openai:chat:gpt-4.1-mini # or -nano variant

Specific snapshot versions are also available:

providers:
  - id: openai:chat:gpt-4.1-2025-04-14 # Standard
  - id: openai:chat:gpt-4.1-mini-2025-04-14 # Mini
  - id: openai:chat:gpt-4.1-nano-2025-04-14 # Nano

GPT-5.1

GPT-5.1 is a GPT-5 family model that emphasizes coding, agentic tasks, and more steerable output behavior.

Available Models

The GPT-5.1 Codex variants are Responses-only models. Use openai:responses:<model> (or the bare openai:<model> shorthand) instead of Chat Completions.

Key Features

GPT-5.1 introduces several improvements over GPT-5:

• none reasoning mode: New lowest reasoning setting for low-latency interactions (default setting)
• Increased steerability: Better control over personality, tone, and output format
• Configurable verbosity: Control output length with low, medium, or high settings (default: medium)

Usage Examples

Fast, low-latency responses:

providers:
  - id: openai:responses:gpt-5.1
    config:
      reasoning:
        effort: 'none' # Default setting - no reasoning tokens
      verbosity: 'low' # Concise outputs

Complex coding and reasoning tasks:

providers:
  - id: openai:responses:gpt-5.1
    config:
      reasoning:
        effort: 'high' # Maximum reasoning for complex tasks
      verbosity: 'medium' # Balanced output length
      max_output_tokens: 4096

Reasoning Modes

GPT-5.1 supports four reasoning effort levels:

• none (default): No reasoning tokens, fastest responses, similar to non-reasoning models
• low: Minimal reasoning for straightforward tasks
• medium: Balanced reasoning for moderate complexity
• high: Maximum reasoning for complex problem-solving

Migration from GPT-5

GPT-5.1 with default settings (none reasoning) is designed as a drop-in replacement for GPT-5. Key differences:

• GPT-5.1 defaults to none reasoning effort (GPT-5 defaulted to low)
• GPT-5.1 has better-calibrated reasoning token consumption
• Improved instruction-following and output formatting

For tasks requiring reasoning, start with medium effort and increase to high if needed.

GPT-5.1-Codex-Max

GPT-5.1-Codex-Max is OpenAI's frontier agentic coding model, built on an updated foundational reasoning model trained on agentic tasks across software engineering, math, research, and more. It's designed for long-running, detailed coding work.

Key Capabilities

• Compaction: First model natively trained to operate across multiple context windows through compaction, coherently working over millions of tokens in a single task
• Long-running tasks: Supports project-scale refactors, deep debugging sessions, and multi-hour agent loops
• Token efficiency: 30% fewer thinking tokens compared to GPT-5.1-Codex at the same reasoning effort level
• Windows support: First model trained to operate in Windows environments
• Improved collaboration: Better performance as a coding partner in CLI environments

Usage Examples

providers:
  - id: openai:responses:gpt-5.1-codex-max
    config:
      reasoning:
        effort: 'medium' # Recommended for most tasks
      max_output_tokens: 25000 # Reserve space for reasoning and outputs

For latency-insensitive tasks requiring maximum quality:

providers:
  - id: openai:responses:gpt-5.1-codex-max
    config:
      reasoning:
        effort: 'xhigh' # Extra high reasoning for best results
      max_output_tokens: 40000

:::warning GPT-5.1-Codex-Max is only available through the Responses API (openai:responses:). It does not work with the Chat Completions API (openai:chat:). :::

Reasoning Effort Levels

• low: Minimal reasoning for straightforward tasks
• medium: Balanced reasoning, recommended as daily driver
• high: Maximum reasoning for complex problem-solving
• xhigh: Extra high reasoning for non-latency-sensitive tasks requiring best results

Best Practices

• Use for agentic coding tasks in Codex or Codex-like environments
• Reserve at least 25,000 tokens for reasoning and outputs when starting
• Start with medium reasoning effort for most tasks
• Use xhigh effort only for complex tasks where latency is not a concern
• Review agent work before deploying to production

:::note GPT-5.1-Codex-Max is recommended for use only in agentic coding environments and is not a general-purpose model like GPT-5.1. :::

GPT-5.2

GPT-5.2 is a GPT-5 family model for coding and agentic tasks, with both standard and pro variants.

Available Models

Key Specifications

• Context window: 400,000 tokens
• Max output tokens: 128,000 tokens
• Reasoning support: Full reasoning token support with configurable effort levels
• Pricing (gpt-5.2, gpt-5.2-chat-latest, gpt-5.2-codex): $1.75 per 1M input tokens, $14 per 1M output tokens
• Pricing (gpt-5.2-pro): $21 per 1M input tokens, $168 per 1M output tokens

Usage Examples

Standard GPT-5.2 and its chat alias are available via both the Chat Completions API and Responses API. GPT-5.2 Codex and Pro are Responses-only:

Chat Completions API:

providers:
  - id: openai:chat:gpt-5.2-chat-latest
    config:
      max_completion_tokens: 4096

  # With reasoning effort
  - id: openai:chat:gpt-5.2
    config:
      reasoning_effort: 'medium'
      max_completion_tokens: 4096

Responses API:

providers:
  - id: openai:responses:gpt-5.2-codex
    config:
      max_output_tokens: 4096

  # With reasoning effort (nested format)
  - id: openai:responses:gpt-5.2
    config:
      reasoning:
        effort: 'medium'
      max_output_tokens: 4096

Fast, low-latency responses (no reasoning):

providers:
  # Chat API
  - id: openai:chat:gpt-5.2
    config:
      reasoning_effort: 'none'
      max_completion_tokens: 2048

  # Responses API
  - id: openai:responses:gpt-5.2
    config:
      reasoning:
        effort: 'none'
      max_output_tokens: 2048

GPT-5.2-pro (including dated snapshots) is Responses-only and supports medium, high, and xhigh reasoning effort; none and low are rejected by the API:

providers:
  - id: openai:responses:gpt-5.2-pro
    config:
      max_output_tokens: 8192
      reasoning:
        effort: 'high'

Key Improvements over GPT-5.1

• Reduced deception: Significantly lower deception rates in production traffic
• Better safety compliance: Improved cyber safety policy compliance
• Improved prompt injection resistance: Enhanced robustness to known prompt injection attacks
• Enhanced sensitive topic handling: Better performance on mental health and emotional reliance evaluations

Reasoning Effort Levels

• none: No reasoning tokens, fastest responses
• low: Minimal reasoning for straightforward tasks
• medium: Balanced reasoning for moderate complexity
• high: Maximum reasoning for complex problem-solving

GPT-5.3 Instant

GPT-5.3 Instant is exposed as gpt-5.3-chat-latest. Promptfoo also supports GPT-5.3 coding variants for agentic/code workflows.

Available Models

Key Specifications

• Endpoint support: gpt-5.3-chat-latest supports Chat Completions and Responses; gpt-5.3-codex is Responses-only.
• Limits and pricing: The -latest alias can move over time. Check OpenAI model docs and API pricing for current context limits and rates.

Usage Examples

providers:
  - id: openai:chat:gpt-5.3-chat-latest
    config:
      max_completion_tokens: 2048

  - id: openai:responses:gpt-5.3-codex
    config:
      reasoning:
        effort: 'high'
      max_output_tokens: 4096

  - id: openai:responses:gpt-5.3-chat-latest
    config:
      max_output_tokens: 2048

<a id="gpt-56-limited-preview"></a>

GPT-5.6

OpenAI launched GPT-5.6 for general availability with three tiers. The gpt-5.6 alias routes to gpt-5.6-sol; use gpt-5.6-terra for balanced cost and performance or gpt-5.6-luna for efficient, high-volume workloads. All three tiers support Chat Completions, Responses, and Batch.

For compatibility with preview-era configs, bare openai:gpt-5.6-sol, -terra, and -luna IDs continue to use Responses. The new bare openai:gpt-5.6 alias uses Chat Completions. Prefer the explicit openai:chat: or openai:responses: prefix when endpoint selection matters.

GPT-5.6 supports max reasoning and reasoning.mode: pro across Sol, Terra, and Luna. Codex ultra is available for Sol and Terra through the Codex SDK or Codex app-server provider as a multi-agent mode, not a Responses API reasoning value.

Prompt-cache reads receive a 90% discount, and cache writes cost 1.25 times the input rate. Promptfoo applies both when the API returns cached_tokens and cache_write_tokens; if a compatible gateway omits cache-write usage, Promptfoo leaves GPT-5.6 cost unset rather than underestimating it. Standard, Batch, and Flex requests above 272,000 input tokens use 2x input and 1.5x output pricing for the entire request; Priority processing does not support long-context requests. Regional processing endpoints add a 10% uplift. Each tier has a 1,050,000-token context window and 128,000 maximum output tokens.

providers:
  - id: openai:responses:gpt-5.6
    config:
      reasoning:
        effort: 'max'
      max_output_tokens: 8192

  - id: openai:responses:gpt-5.6-terra
    config:
      reasoning:
        effort: 'medium'
      max_output_tokens: 4096

  - id: openai:responses:gpt-5.6-luna
    config:
      reasoning:
        effort: 'low'
      max_output_tokens: 2048

GPT-5.5

GPT-5.5 is a high-capability GPT-5 family model for professional work and agentic workflows.

Available Models

Key Specifications

• Long-context pricing: gpt-5.5 uses $10.00 input / $45.00 output per 1M tokens when prompts exceed 272,000 input tokens.
• Context window: gpt-5.5 and gpt-5.5-pro support 1,050,000 tokens.
• Max output tokens: 128,000 tokens.
• Reasoning effort: gpt-5.5 supports none, low, medium, high, and xhigh. In Chat Completions, set reasoning_effort; in Responses API, set reasoning.effort.
• Endpoint support: gpt-5.5 supports Chat Completions and Responses API. gpt-5.5-pro is Responses API only and supports Batch API.
• Cached input: gpt-5.5 cached input tokens are $0.50 per 1M. gpt-5.5-pro has no cached-input discount.
• Cost estimates: Promptfoo uses returned usage metadata for GPT-5.5 pricing and infers Batch, Flex, or Priority rates when the API response or configured service_tier identifies that tier.
• Long-running requests: gpt-5.5-pro automatically receives the same 10-minute timeout as other GPT-5 pro models.

Usage Examples

providers:
  - id: openai:chat:gpt-5.5
    config:
      max_completion_tokens: 4096
      reasoning_effort: 'low'
      verbosity: 'medium'

  - id: openai:responses:gpt-5.5
    config:
      reasoning:
        effort: 'high'
      max_output_tokens: 4096

  - id: openai:responses:gpt-5.5-pro
    config:
      reasoning:
        effort: 'xhigh'
      max_output_tokens: 8192

GPT-5.4

GPT-5.4 is a GPT-5 family model for complex professional work, agentic coding, and tool-heavy workflows.

Available Models

Key Specifications

• Context window: gpt-5.4 and gpt-5.4-pro support 1,050,000 tokens. gpt-5.4-mini and gpt-5.4-nano support 400,000 tokens.
• Long-context pricing: gpt-5.4 and gpt-5.4-pro use higher long-context rates when prompts exceed 272,000 input tokens.
• Max output tokens: 128,000 tokens
• Reasoning effort: gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano support none, low, medium, high, xhigh. gpt-5.4-pro supports medium, high, xhigh.
• Endpoint support: gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano support Chat Completions and Responses API. gpt-5.4-pro is Responses API only. Promptfoo's Codex SDK provider supports gpt-5.4, gpt-5.4-pro, and the newer GPT-5.5 line.
• Cached input: gpt-5.4 cached input tokens $0.25 per 1M, gpt-5.4-mini $0.075 per 1M, and gpt-5.4-nano $0.02 per 1M. gpt-5.4-pro has no cached-input discount.

Usage Examples

providers:
  - id: openai:chat:gpt-5.4-mini
    config:
      max_completion_tokens: 2048
      reasoning_effort: 'none'
      verbosity: 'low'

  - id: openai:chat:gpt-5.4
    config:
      max_completion_tokens: 4096
      reasoning_effort: 'low'

  - id: openai:responses:gpt-5.4-nano
    config:
      reasoning:
        effort: 'none'
      max_output_tokens: 1024

  - id: openai:responses:gpt-5.4
    config:
      reasoning:
        effort: 'high'
      max_output_tokens: 4096

  - id: openai:responses:gpt-5.4-mini
    config:
      reasoning:
        effort: 'medium'
      max_output_tokens: 4096

  - id: openai:responses:gpt-5.4-pro
    config:
      reasoning:
        effort: 'xhigh'
      max_output_tokens: 8192

Reasoning Models (o1, o3, o3-pro, o3-mini, o4-mini)

Reasoning models, like o1, o3, o3-pro, o3-mini, and o4-mini, are large language models trained with reinforcement learning to perform complex reasoning. These models excel in complex problem-solving, coding, scientific reasoning, and multi-step planning for agentic workflows.

When using reasoning models, there are important differences in how tokens are handled:

providers:
  - id: openai:o1
    config:
      reasoning:
        effort: 'medium' # Can be "low", "medium", or "high"
      max_completion_tokens: 25000 # Can also be set via OPENAI_MAX_COMPLETION_TOKENS env var

Unlike standard models that use max_tokens, reasoning models use:

• max_completion_tokens to control the total tokens generated (both reasoning and visible output)
• reasoning to control how thoroughly the model thinks before responding (with effort: none, low, medium, high; some GPT-5 family models also support minimal or xhigh)

How Reasoning Models Work

Reasoning models "think before they answer," generating internal reasoning tokens that:

• Are not visible in the output
• Count towards token usage and billing
• Occupy space in the context window

Both o1 and o3-mini models have a 128,000 token context window, while o3-pro and o4-mini have a 200,000 token context window. OpenAI recommends reserving at least 25,000 tokens for reasoning and outputs when starting with these models.

Images

Sending images in prompts

You can include images in the prompt by using content blocks. For example, here's an example config:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - file://prompt.json

providers:
  - openai:gpt-5

tests:
  - vars:
      question: 'What do you see?'
      url: 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg'
  # ...

And an example prompt.json:

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "{{question}}"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "{{url}}"
        }
      }
    ]
  }
]

See the OpenAI vision example.

Generating images

OpenAI supports image generation via openai:image:<model>. Supported models include:

• gpt-image-2 - OpenAI's latest image generation model with flexible custom sizes
• gpt-image-1.5 - High-quality GPT Image model with strong instruction following
• gpt-image-1 - High-quality image generation model
• gpt-image-1-mini - Cost-efficient version of GPT Image 1

dall-e-3 and dall-e-2 were retired from the OpenAI API on May 12, 2026. Promptfoo retains request compatibility for gateways that still expose them; use a current GPT Image model for new evals.

The openai:image provider uses the Image API generations endpoint. It supports text-to-image generation; image edit/reference inputs (image, mask, input_fidelity), streaming (stream/partial_images), and variations are not implemented in this provider.

See the OpenAI image generation example.

GPT Image 2

GPT Image 2 is OpenAI's latest image generation model. It supports the standard GPT Image output controls plus custom sizes that satisfy OpenAI's dimensional constraints.

providers:
  - id: openai:image:gpt-image-2
    config:
      size: 1024x1024 # auto, common sizes, or custom WIDTHxHEIGHT
      quality: low # low, medium, high, or auto
      background: opaque # opaque or auto
      output_format: webp # png, jpeg, or webp
      output_compression: 80 # 0-100, only set with jpeg/webp
      moderation: auto # auto or low
      n: 1 # 1-10 images
      user: promptfoo-user # optional end-user identifier

For custom size values, both dimensions must be multiples of 16, the maximum edge must be no larger than 3840px, the long edge to short edge ratio must be at most 3:1, and total pixels must be between 655,360 and 8,294,400.

Pricing:

These are output image estimates. Input text tokens may also apply, and OpenAI may return usage data for the request. For GPT Image 2 quality: auto, omitted quality, or custom sizes, promptfoo leaves cost unset and preserves the returned usage in tokenUsage/metadata.usage instead of guessing.

GPT Image 1.5

GPT Image 1.5 is a high-quality image generation model with strong instruction following, prompt adherence, and photorealistic quality. It uses token-based pricing for more flexible cost control.

providers:
  - id: openai:image:gpt-image-1.5
    config:
      size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, or auto
      quality: low # low, medium, high, or auto
      background: transparent # transparent, opaque, or auto
      output_format: webp # png, jpeg, or webp
      output_compression: 80 # 0-100, only set with jpeg/webp
      moderation: auto # auto or low

Pricing:

GPT Image 1.5 uses token-based pricing at $5/1M input text tokens, $10/1M output text tokens, $8/1M input image tokens, and $32/1M output image tokens. Estimated costs per image:

chatgpt-image-latest is also supported through openai:image:chatgpt-image-latest. It uses the same generation parameters and per-image/token rates as GPT Image 1.5; OpenAI recommends GPT Image 2 for new integrations. Promptfoo currently supports image generation, not the Images API edit or inpainting operations.

GPT Image 1

GPT Image 1 is a high-quality image generation model with superior instruction following, text rendering, and real-world knowledge.

providers:
  - id: openai:image:gpt-image-1
    config:
      size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, or auto
      quality: low # low, medium, high, or auto
      background: transparent # transparent, opaque, or auto
      output_format: webp # png, jpeg, or webp
      output_compression: 80 # 0-100, only set with jpeg/webp
      moderation: auto # auto or low

Pricing:

GPT Image 1 Mini

GPT Image 1 Mini is a cost-efficient version of GPT Image 1 with the same capabilities at lower cost.

providers:
  - id: openai:image:gpt-image-1-mini
    config:
      size: 1024x1024 # 1024x1024, 1024x1536, 1536x1024, or auto
      quality: low # low, medium, high, or auto
      background: transparent # transparent, opaque, or auto
      output_format: webp # png, jpeg, or webp
      output_compression: 80 # 0-100, only set with jpeg/webp
      moderation: auto # auto or low

Pricing:

Example

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'In the style of Van Gogh: {{subject}}'
  - 'In the style of Dali: {{subject}}'

providers:
  - openai:image:gpt-image-2

tests:
  - vars:
      subject: bananas
  - vars:
      subject: new york city

To display images in the web viewer, wrap vars or outputs in markdown image tags like so:

![](/path/to/myimage.png)

Then, enable 'Render markdown' under Table Settings.

Video Generation (Sora)

OpenAI supports video generation via openai:video:<model>. Supported models and snapshots include:

• sora-2, sora-2-2025-10-06, sora-2-2025-12-08 - standard video generation
• sora-2-pro, sora-2-pro-2025-10-06 - higher-quality video generation

The Sora Videos API is scheduled for removal on September 24, 2026. sora-2-pro-high-res is a pricing tier, not a separate model ID; select a supported Pro resolution instead.

Basic Usage

providers:
  - id: openai:video:sora-2
    config:
      size: 1280x720 # Standard supports 1280x720 and 720x1280
      seconds: 8 # Duration: 4, 8, 12, 16, or 20 seconds

Configuration Options

Example Configuration

prompts:
  - 'A cinematic shot of: {{scene}}'

providers:
  - id: openai:video:sora-2
    config:
      size: 1280x720
      seconds: 4
  - id: openai:video:sora-2-pro
    config:
      size: 720x1280
      seconds: 8

tests:
  - vars:
      scene: a cat riding a skateboard through a city
  - vars:
      scene: waves crashing on a beach at sunset

Image-to-Video Generation

Generate videos starting from a source image using input_reference:

providers:
  - id: openai:video:sora-2
    config:
      input_reference: file://assets/start-image.png
      seconds: 4

prompts:
  - 'Animate this image: the character slowly walks forward'

The input_reference accepts a file:// path, image URL, base64-encoded image data, data URL, or an uploaded-file reference such as { file_id: file_123 }. Promptfoo sends the documented { image_url: ... } or { file_id: ... } JSON shape to OpenAI; object references must provide exactly one of image_url or file_id.

Reusable Sora characters can be supplied with characters: [{ id: char_123 }]; at most two characters can be used in one generation, and the character name should also appear in the prompt.

Credential-bearing image URLs and authenticated custom video gateways always bypass the persistent video cache. Promptfoo does not place authentication headers or their fingerprints in long-lived cache keys. Prompts, tenant headers, and gateway URL paths containing credentials also bypass persistent caching.

Sora characters and file_id image references are project-scoped resources. Promptfoo only caches their generated videos when an explicit, non-secret project or tenant header such as OpenAI-Project or X-Tenant-Id is configured; OpenAI-Organization alone does not isolate projects.

Provider and per-prompt headers are sent for video creation, status polling, and content downloads so routed gateways can authorize the full job lifecycle.

Video Remixing (Legacy)

Remix an existing Sora video with a new prompt using remix_video_id:

providers:
  - id: openai:video:sora-2
    config:
      remix_video_id: video_abc123def456

prompts:
  - 'Make the scene more dramatic with stormy weather'

The remix_video_id is the video ID returned from a previous Sora generation (found in response.video.id). Remixes inherit the source video's model, size, and duration; creation-only options such as size, seconds, input_reference, and characters are ignored. Promptfoo reports cost and metadata using the completed remix job returned by OpenAI.

:::note Remixed videos are not cached since each remix produces unique results even with the same prompt. :::

OpenAI has deprecated the remix endpoint and recommends POST /v1/videos/edits for new integrations. Promptfoo currently exposes the legacy remix_video_id path and does not implement the newer Videos API edits or extensions endpoints.

Viewing Generated Videos

Videos are automatically displayed in the web viewer with playback controls. The viewer shows:

• Video player with controls
• Video metadata (model, size, duration)
• Thumbnail preview (if enabled)

Videos are stored in promptfoo's media storage (~/.promptfoo/media/) and served via the web interface.

Pricing

Sora Pro supports 1080p creation and remixes at the published $0.70-per-second rate. Promptfoo reports the rate using the completed video's returned resolution.

Web Search Support

The OpenAI Responses API supports both the standard web_search tool and the web_search_preview tool family. Chat Completions also supports built-in search models such as gpt-5-search-api. All three forms enable the search-rubric assertion type. Deep research models require at least one supported data source, which can be either web-search tool, file search with a vector store, or a remote MCP server. These tools let models search the web for current information and verify facts.

Enabling Web Search

To enable web search with the OpenAI Responses API, use the openai:responses provider format and add either the standard web_search tool or the preview web_search_preview tool to your configuration:

providers:
  - id: openai:responses:gpt-5.1
    config:
      tools:
        - type: web_search

To preserve an existing Chat Completions search integration, use a built-in search model. These models retrieve information before each response and return URL citations in response.metadata.annotations:

providers:
  - id: openai:chat:gpt-5-search-api
    config:
      passthrough:
        web_search_options:
          search_context_size: high

Using Web Search Assertions

The search-rubric assertion type uses web search to quickly verify current information:

• Real-time data (weather, stock prices, news)
• Current events and statistics
• Time-sensitive information
• Quick fact verification

Example configuration:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'What is the current temperature in {{city}}?'

providers:
  - id: openai:responses:gpt-5.1
    config:
      tools:
        - type: web_search_preview

tests:
  - vars:
      city: New York
    assert:
      - type: search-rubric
        value: Current temperature in New York City

Cost Considerations

:::info Web search calls are billed separately from normal tokens:

• The web search tool costs $10 per 1,000 calls for the standard tool and $10-25 per 1,000 calls for preview variants, plus any search content tokens where applicable
• Chat Completions gpt-5-search-api costs $10 per 1,000 calls plus tokens; the older GPT-4o search-preview models cost $25 per 1,000 calls plus tokens
• Only web_search_call.action.type: search incurs a search fee; open_page and find_in_page are observable actions but are not charged as separate searches
• Each search-rubric assertion may perform one or more searches
• Caching is enabled by default; use --no-cache to force fresh searches during development
• See OpenAI's pricing page for current rates :::

Best Practices

1. Use specific search queries: More specific queries yield better verification results
2. Use caching: Caching is enabled by default; results are reused to avoid repeated searches
3. Use appropriate models: gpt-5.4-mini is a cost-effective option for web search
4. Monitor usage: Track API costs, especially in CI/CD pipelines

For more details on using search-rubric assertions, see the Search-Rubric documentation.

Tool Calling

Using tools

To set tools on an OpenAI provider, use the provider's config key. The model may return tool calls in two formats:

1. An array of tool calls: [{type: 'function', function: {...}}]
2. A message with tool calls: {content: '...', tool_calls: [{type: 'function', function: {...}}]}

Tools can be defined inline or loaded from an external file:

:::info Supported file formats

Tools can be loaded from external files in multiple formats:

# Static data files (choose one)
tools: file://./tools.yaml
# tools: file://./tools.json

# Dynamic tool definitions from code (choose one; requires a function name)
# tools: file://./tools.py:get_tools
# tools: file://./tools.js:getTools
# tools: file://./tools.ts:getTools

Python and JavaScript files must export a function that returns the tool definitions array. The function can be synchronous or asynchronous.

Asynchronous example:

// tools.js - Fetch tool definitions from API at runtime
export async function getTools() {
  const apiKey = process.env.INTERNAL_API_KEY;
  const response = await fetch('https://api.internal.com/tool-definitions', {
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  const tools = await response.json();
  return tools;
}

:::

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - file://prompt.txt
providers:
  - id: openai:chat:gpt-5.4-mini
    // highlight-start
    config:
      tools:
        - type: function
          function:
            name: get_current_weather
            description: Get the current weather in a given location
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: The city and state, e.g. San Francisco, CA
                unit:
                  type: string
                  enum: [celsius, fahrenheit]
              required: [location]
      tool_choice: 'auto'
    // highlight-end

tests:
  - vars:
      city: Boston
    assert:
      - type: is-json
      - type: is-valid-openai-tools-call
      - type: javascript
        value: output[0].function.name === 'get_current_weather'
      - type: javascript
        value: JSON.parse(output[0].function.arguments).location === 'Boston, MA'

  - vars:
      city: New York
# ...

Sometimes OpenAI function calls don't match tools schemas. Use an is-valid-openai-tools-call assertion to enforce an exact schema match between tools and the function definition.

To further test tools definitions, you can use the javascript assertion and/or transform directives. For example:

tests:
  - vars:
      city: Boston
    assert:
      - type: is-json
      - type: is-valid-openai-tools-call
      - type: javascript
        value: output[0].function.name === 'get_current_weather'
      - type: javascript
        value: JSON.parse(output[0].function.arguments).location === 'Boston, MA'

  - vars:
      city: New York
      # transform returns only the 'name' property
    transform: output[0].function.name
    assert:
      - type: is-json
      - type: similar
        value: NYC

:::tip Functions can use variables from test cases:

{
  type: "function",
  function: {
    description: "Get temperature in {{city}}"
    // ...
  }
}

They can also include functions that dynamically reference vars:

{
  type: "function",
  function: {
    name: "get_temperature",
    parameters: {
      type: "object",
        properties: {
          unit: {
            type: "string",
            // highlight-start
            enum: (vars) => vars.units,
            // highlight-end
          }
        },
    }
  }
}

:::

Using functions

functions and function_call are deprecated in favor of tools and tool_choice. See the OpenAI API reference.

Use the functions config to define custom functions. Each function should be an object with a name, optional description, and parameters. For example:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - file://prompt.txt
providers:
  - id: openai:chat:gpt-5.4-mini
    // highlight-start
    config:
      functions:
        [
          {
            'name': 'get_current_weather',
            'description': 'Get the current weather in a given location',
            'parameters':
              {
                'type': 'object',
                'properties':
                  {
                    'location':
                      {
                        'type': 'string',
                        'description': 'The city and state, e.g. San Francisco, CA',
                      },
                    'unit': { 'type': 'string', 'enum': ['celsius', 'fahrenheit'] },
                  },
                'required': ['location'],
              },
          },
        ]
    // highlight-end
tests:
  - vars:
      city: Boston
    assert:
      // highlight-next-line
      - type: is-valid-openai-function-call
  - vars:
      city: New York
  # ...

Sometimes OpenAI function calls don't match functions schemas. Use is-valid-openai-function-call assertions to enforce an exact schema match between function calls and the function definition.

To further test function call definitions, you can use the javascript assertion and/or transform directives. For example:

tests:
  - vars:
      city: Boston
    assert:
      - type: is-valid-openai-function-call
      - type: javascript
        value: output.name === 'get_current_weather'
      - type: javascript
        value: JSON.parse(output.arguments).location === 'Boston, MA'

  - vars:
      city: New York
    # transform returns only the 'name' property for this test case
    transform: output.name
    assert:
      - type: is-json
      - type: similar
        value: NYC

Loading tools/functions from a file

Instead of duplicating function definitions across multiple configurations, you can reference an external YAML (or JSON) file that contains your functions. This allows you to maintain a single source of truth for your functions, which is particularly useful if you have multiple versions or regular changes to definitions.

:::tip

Tool definitions can be loaded from JSON, YAML, Python, or JavaScript files. For Python/JS files, specify a function name that returns the tool definitions: file://tools.py:get_tools

:::

To load your functions from a file, specify the file path in your provider configuration like so:

providers:
  - file://./path/to/provider_with_function.yaml

You can also use a pattern to load multiple files:

providers:
  - file://./path/to/provider_*.yaml

Here's an example of how your provider_with_function.yaml might look:

id: openai:chat:gpt-5.4-mini
config:
  functions:
    - name: get_current_weather
      description: Get the current weather in a given location
      parameters:
        type: object
        properties:
          location:
            type: string
            description: The city and state, e.g. San Francisco, CA
          unit:
            type: string
            enum:
              - celsius
              - fahrenheit
            description: The unit in which to return the temperature
        required:
          - location

Using response_format

Promptfoo supports the response_format parameter, which allows you to specify the expected output format.

response_format can be included in the provider config, or in the prompt config.

Prompt config example

prompts:
  - label: 'Prompt #1'
    raw: 'You are a helpful math tutor. Solve {{problem}}'
    config:
      response_format:
        type: json_schema
        json_schema: ...

Provider config example

providers:
  - id: openai:chat:gpt-5.4-mini
    config:
      response_format:
        type: json_schema
        json_schema: ...

External file references

To make it easier to manage large JSON schemas, external file references are supported for response_format in both Chat and Responses APIs. This is particularly useful for:

• Reusing complex JSON schemas across multiple configurations
• Managing large schemas in separate files for better organization
• Version controlling schemas independently from configuration files

config:
  response_format: file://./path/to/response_format.json

The external file should contain the complete response_format configuration object. Chat Completions uses a nested json_schema object:

{
  "type": "json_schema",
  "json_schema": {
    "name": "event_extraction",
    "schema": {
      "type": "object",
      "properties": {
        "event_name": { "type": "string" },
        "date": { "type": "string" },
        "location": { "type": "string" }
      },
      "required": ["event_name", "date", "location"],
      "additionalProperties": false
    },
    "strict": true
  }
}

The Responses API uses the flattened format:

{
  "type": "json_schema",
  "name": "event_extraction",
  "schema": {
    "type": "object",
    "properties": {
      "event_name": { "type": "string" },
      "date": { "type": "string" },
      "location": { "type": "string" }
    },
    "required": ["event_name", "date", "location"],
    "additionalProperties": false
  }
}

You can also use nested file references for the schema itself, which is useful for sharing schemas across multiple response formats:

{
  "type": "json_schema",
  "name": "event_extraction",
  "schema": "file://./schemas/event-schema.json"
}

Variable rendering is supported in file paths using Nunjucks syntax:

config:
  response_format: file://./schemas/{{ schema_name }}.json

For a complete example with the Chat API, see the OpenAI Structured Output example or initialize it with:

npx promptfoo@latest init --example openai-structured-output

For an example with the Responses API, see the OpenAI Responses API example and run:

npx promptfoo@latest init --example openai-responses
cd openai-responses
npx promptfoo@latest eval -c promptfooconfig.external-format.yaml

Per-test structured output

You can use different JSON schemas for different test cases using the test.options field. This allows a single prompt to produce different structured output formats depending on the test:

prompts:
  - 'Answer this question: {{question}}'

providers:
  - openai:gpt-4o-mini

# Parse JSON output so assertions can access properties directly
defaultTest:
  options:
    transform: JSON.parse(output)

tests:
  # Math problems use math schema
  - vars:
      question: 'What is 15 * 7?'
    options:
      response_format: file://./schemas/math-response-format.json
    assert:
      - type: javascript
        value: output.answer === 105

  # Comparison questions use comparison schema
  - vars:
      question: 'Compare apples and oranges'
    options:
      response_format: file://./schemas/comparison-response-format.json
    assert:
      - type: javascript
        value: output.winner === 'item1' || output.winner === 'item2' || output.winner === 'tie'

Each schema file contains the complete response_format object. See the per-test schema example for a full working configuration.

Supported environment variables

These OpenAI-related environment variables are supported:

Evaluating assistants

:::warning The Assistants API is deprecated and scheduled to shut down on August 26, 2026. For new integrations, use the Responses API and see OpenAI's Assistants migration guide. :::

To test out an Assistant via OpenAI's Assistants API, first create an Assistant in the API playground.

Set functions, code interpreter, and files for file search as necessary.

Then, include the assistant in your config:

prompts:
  - 'Write a tweet about {{topic}}'
providers:
  - openai:assistant:asst_fEhNN3MClMamLfKLkIaoIpgZ
tests:
  - vars:
      topic: bananas
  # ...

Code interpreter, function calls, and file-search results will be included in the output alongside chat messages. The evaluator creates a new thread for each eval.

The following properties can be overwritten in provider config:

• modelName - OpenAI model to use
• instructions - System prompt
• tools - Enabled tools
• temperature - Temperature for the model
• toolChoice - Controls whether the AI should use a tool
• tool_resources - Tool resources to include in the thread - see Assistant v2 tool resources
• attachments - File attachments to include in messages - see Assistant v2 attachments

Here's an example of a more detailed config:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'Write a tweet about {{topic}}'
providers:
  // highlight-start
  - id: openai:assistant:asst_fEhNN3MClMamLfKLkIaoIpgZ
    config:
      modelName: gpt-4.1
      instructions: "You always speak like a pirate"
      temperature: 0.2
      toolChoice:
        type: file_search
      tools:
        - type: code_interpreter
        - type: file_search
  // highlight-end
tests:
  - vars:
      topic: bananas
  # ...

Automatically handling function tool calls

You can specify JavaScript callbacks that are automatically called to create the output of a function tool call.

This requires defining your config in a JavaScript file instead of YAML.

module.exports = /** @type {import('promptfoo').TestSuiteConfig} */ ({
  prompts: 'Please add the following numbers together: {{a}} and {{b}}',
  providers: [
    {
      id: 'openai:assistant:asst_fEhNN3MClMamLfKLkIaoIpgZ',
      config: {
        modelName: 'gpt-4.1',
        instructions: 'You can add two numbers together using the `addNumbers` tool',
        tools: [
          {
            type: 'function',
            function: {
              name: 'addNumbers',
              description: 'Add two numbers together',
              parameters: {
                type: 'object',
                properties: {
                  a: { type: 'number' },
                  b: { type: 'number' },
                },
                required: ['a', 'b'],
                additionalProperties: false,
              },
              strict: true,
            },
          },
        ],
        /**
         * Map of function tool names to function callback.
         */
        functionToolCallbacks: {
          // this function should accept a JSON-parsed value, and return a string
          // or a `Promise<string>`.
          addNumbers: (parameters) => {
            const { a, b } = parameters;
            return JSON.stringify(a + b);
          },
        },
      },
    },
  ],
  tests: [
    {
      vars: { a: 5, b: 6 },
    },
  ],
});

Audio capabilities

OpenAI models with audio support (like gpt-audio-1.5, gpt-audio, and gpt-audio-mini) can process audio inputs and generate audio outputs. This enables testing speech-to-text, text-to-speech, and speech-to-speech capabilities.

Use these general-purpose audio models with openai:chat:*. Current OpenAI docs place gpt-audio* on the Chat Completions endpoint, and audio is not yet supported by the Responses API, so openai:responses:gpt-audio* is not a valid audio route.

OpenAI retired the older gpt-4o-audio-preview model family on May 7, 2026. Use the current gpt-audio* models below for new evals.

Available audio models:

• gpt-audio-1.5 - Flagship audio model ($2.50/$10 per 1M text tokens, $32/$64 per 1M audio tokens)
• gpt-audio - General audio model ($2.50/$10 per 1M text tokens, $32/$64 per 1M audio tokens)
• gpt-audio-2025-08-28 - Dated snapshot of gpt-audio
• gpt-audio-mini - Cost-efficient audio model ($0.60/$2.40 per 1M text tokens, $10/$20 per 1M audio tokens)
• gpt-audio-mini-2025-12-15 - Dated snapshot of gpt-audio-mini
• gpt-audio-mini-2025-10-06 - Dated snapshot of gpt-audio-mini

Using audio inputs

You can include audio files in your prompts using the following format:

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "You are a helpful customer support agent. Listen to the customer's request and respond with a helpful answer."
      },
      {
        "type": "input_audio",
        "input_audio": {
          "data": "{{audio_file}}",
          "format": "mp3"
        }
      }
    ]
  }
]

With a corresponding configuration:

prompts:
  - id: file://audio-input.json
    label: Audio Input

providers:
  - id: openai:chat:gpt-audio-1.5
    config:
      modalities: ['text'] # also supports 'audio'

tests:
  - vars:
      audio_file: file://assets/transcript1.mp3
    assert:
      - type: llm-rubric
        value: Resolved the customer's issue

Supported audio file formats include WAV, MP3, OGG, AAC, M4A, and FLAC.

Audio configuration options

The audio configuration supports these parameters:

In the web UI, audio outputs display with an embedded player and transcript. For a complete working example, see the OpenAI audio example or initialize it with:

npx promptfoo@latest init --example openai-audio

Text-to-speech

Use openai:tts:<model> or openai:speech:<model> to generate playable audio through /v1/audio/speech. Supported models are gpt-4o-mini-tts, gpt-4o-mini-tts-2025-12-15, gpt-4o-mini-tts-2025-03-20, tts-1, tts-1-1106, tts-1-hd, and tts-1-hd-1106. The March 2025 mini snapshot is scheduled for removal on July 23, 2026; prefer the current alias or December snapshot for new evals.

prompts:
  - 'Welcome to the evaluation.'

providers:
  - id: openai:tts:gpt-4o-mini-tts
    config:
      voice: coral
      instructions: Speak warmly and clearly.
      response_format: wav # mp3, opus, aac, flac, wav, or pcm
      speed: 1.0 # 0.25-4.0

Speech input is limited to 4,096 characters. instructions is supported by GPT-4o mini TTS, not tts-1 or tts-1-hd. Custom voices can be provided as voice: { id: voice_123 } and support the current language and format fields. format is an alias for response_format: the speech API ignores a top-level format field, so promptfoo sends the value as response_format. If both are set, the explicit response_format takes precedence.

providers:
  - id: openai:tts:gpt-4o-mini-tts
    config:
      voice: { id: voice_123 }
      language: fr
      format: wav # sent to the API as response_format: wav

The legacy models are billed at $15 and $30 per million characters, respectively. GPT-4o mini TTS is billed using input and audio-output tokens; the binary speech response does not expose usage, so promptfoo leaves cost unset instead of estimating it.

Use passthrough to send experimental speech fields or gateway-specific request options that do not yet have a dedicated provider setting.

Custom speech gateways authenticated through headers or URL credentials always bypass the speech cache. Promptfoo does not place authentication headers or their fingerprints in long-lived cache keys, preventing differently authenticated calls from sharing audio. Speech inputs, instructions, passthrough values, tenant headers, and gateway URL paths containing credentials also bypass persistent caching. Header credentials such as authorization or x-api-key satisfy the provider's authentication requirement; a separate OpenAI API key is not needed.

Uploaded custom voices are project-scoped. To cache a custom-voice response, provide an explicit, non-secret project or tenant header such as OpenAI-Project or X-Tenant-Id; OpenAI-Organization alone does not isolate projects.

Audio transcription

OpenAI provides dedicated transcription models for converting speech to text. GPT-4o transcription models report token usage, which promptfoo prices using the published input/output token rates; the per-minute figures below are OpenAI's cost estimates. Whisper is billed per minute.

Available transcription models:

To use transcription models, specify the provider format openai:transcription:<model name>:

prompts:
  - '{{ audio_file }}'

providers:
  - id: openai:transcription:whisper-1
    config:
      language: en # Optional: specify language for better accuracy
      temperature: 0 # Optional: 0 for more deterministic output

  - id: openai:transcription:gpt-4o-transcribe
    config:
      language: en
      prompt: This is a technical discussion about AI and machine learning.

  - id: openai:transcription:gpt-4o-transcribe-diarize
    config:
      chunking_strategy: auto

tests:
  - vars:
      audio_file: path/to/sample-audio.mp3
    assert:
      - type: contains
        value: expected transcript content

Transcription configuration options

Supported audio formats include MP3, MP4, MPEG, MPGA, M4A, WAV, and WEBM. Promptfoo automatically sets chunking_strategy: auto for diarization, which is required for inputs longer than 30 seconds. timestamp_granularities is only sent for whisper-1, and prompt is not supported for diarization. See OpenAI's speech-to-text guide for speaker-reference requirements.

For noisy audio, configure server-side VAD explicitly:

chunking_strategy:
  type: server_vad
  threshold: 0.6
  prefix_padding_ms: 300
  silence_duration_ms: 500

gpt-realtime-whisper is available through the native Realtime transcription-session API rather than the /audio/transcriptions endpoint used by openai:transcription:*. For conversational Realtime evals, use it as input_audio_transcription.model instead.

Diarization example

The diarization model identifies different speakers in the audio:

prompts:
  - '{{ audio_file }}'

providers:
  - id: openai:transcription:gpt-4o-transcribe-diarize
    config:
      chunking_strategy: auto
      known_speaker_names: ['Interviewer', 'Guest']
      # Replace these placeholders with base64-encoded 2-10 second reference clips.
      known_speaker_references:
        - 'data:audio/wav;base64,<interviewer-reference>'
        - 'data:audio/wav;base64,<guest-reference>'

tests:
  - vars:
      audio_file: path/to/interview.mp3
    assert:
      - type: contains
        value: Interviewer
      - type: contains
        value: Guest

For a complete working example, see the OpenAI audio transcription example or initialize it with:

npx promptfoo@latest init --example openai-audio-transcription

Realtime API Models

The Realtime API allows for real-time communication with models like gpt-realtime-2.1, gpt-realtime-2.1-mini, and gpt-realtime-2 using WebSockets, supporting text, audio, and image inputs with streaming text and audio outputs.

Supported Realtime Models

• gpt-realtime-2.1 - Current flagship realtime model ($4/$24 per 1M text tokens, $32/$64 per 1M audio tokens, $5/$0.50 per 1M image/cached-image input tokens)
• gpt-realtime-2.1-mini - Current cost-efficient realtime model ($0.60/$2.40 per 1M text tokens, $10/$20 per 1M audio tokens, $0.80/$0.08 per 1M image/cached-image input tokens)
• gpt-realtime-2 - Reasoning-capable realtime model ($4/$24 per 1M text tokens, $32/$64 per 1M audio tokens)
• gpt-realtime-1.5 - Previous flagship realtime model ($4/$16 per 1M text tokens, $32/$64 per 1M audio tokens)
• gpt-realtime - General-availability realtime model ($4/$16 per 1M text tokens, $32/$64 per 1M audio tokens)
• gpt-realtime-2025-08-28 - Dated snapshot of gpt-realtime
• gpt-realtime-mini - Cost-efficient realtime model ($0.60/$2.40 per 1M text tokens, $10/$20 per 1M audio tokens)
• gpt-realtime-mini-2025-12-15
• gpt-realtime-mini-2025-10-06
• gpt-4o-mini-realtime-preview-2024-12-17 - Deprecated preview snapshot still available until its published shutdown date

Prefer the current gpt-realtime* models for new evals. OpenAI removed several older preview aliases on May 7, 2026, while the dated gpt-4o-mini-realtime-preview-2024-12-17 snapshot remains available until its published July 23, 2026 shutdown date.

The Realtime 2.1 models support a 128k context window and up to 32k output tokens. The current Realtime request schema still accepts an integer max_response_output_tokens only up to 4,096; use 'inf' to allow the model maximum.

Using Realtime API

To use the OpenAI Realtime API, use the provider format openai:realtime:<model name>:

providers:
  - id: openai:realtime:gpt-realtime-1.5
    config:
      modalities: ['text', 'audio']
      voice: 'alloy'
      instructions: 'You are a helpful assistant.'
      websocketTimeout: 60000 # 60 seconds
      # Optional: point to custom/proxy endpoints; WS URL is derived automatically
      # https:// → wss://, http:// → ws://
      # Example: wss://my-custom-api.com/v1/realtime
      # Example: ws://localhost:8080/v1/realtime
      # apiBaseUrl: 'https://my-custom-api.com/v1'

Realtime-specific Configuration Options

The Realtime API configuration supports these parameters in addition to standard OpenAI parameters:

Promptfoo accepts the configuration names above for backward compatibility, then sends the current GA Realtime wire shape to OpenAI: type: 'realtime', native output_modalities, nested audio.input / audio.output, and the documented top-level tool fields.

gpt-realtime-whisper can be used as input_audio_transcription.model inside a conversational Realtime session. Its delay field is supported there; prompt is not supported for that model. OpenAI also exposes gpt-realtime-translate through a separate Realtime translation-session API; that is a different endpoint from the conversational openai:realtime:* provider documented here.

Custom endpoints and proxies (Realtime)

The Realtime provider respects the same base URL configuration as other OpenAI providers. The WebSocket URL is derived from getApiUrl() by converting protocols: https:// → wss:// and http:// → ws://.

You can use this to target Azure-compatible endpoints, proxies, or local/dev servers:

providers:
  - id: openai:realtime:gpt-realtime-1.5
    config:
      apiBaseUrl: 'https://my-custom-api.com/v1' # connects to wss://my-custom-api.com/v1/realtime
      modalities: ['text']

Environment variables OPENAI_API_BASE_URL and OPENAI_BASE_URL also apply to Realtime WebSocket connections.

Function Calling with Realtime API

The Realtime API supports function calling via tools, similar to the Chat API. Here's an example configuration:

Realtime tools can be supplied inline or loaded through the same file:// tool references supported by the rest of the OpenAI provider. The native Realtime format matches the OpenAI docs: function tools use top-level fields, and tool_choice is passed through in the Realtime shape you provide.

providers:
  - id: openai:realtime:gpt-realtime-1.5
    config:
      tools:
        - type: function
          name: get_weather
          description: Get the current weather for a location
          parameters:
            type: object
            properties:
              location:
                type: string
                description: The city and state, e.g. San Francisco, CA
            required: ['location']
      tool_choice: 'auto'

Realtime function tools use top-level name, description, and parameters fields, exactly as shown in the OpenAI Realtime function-calling guide. Native Realtime tool_choice values are sent unchanged.

For compatibility with shared Chat Completions configs, promptfoo also accepts nested Chat-style function tool objects such as type: function plus function: { ... } and converts only those legacy shapes into the native Realtime format before sending them.

When you provide a custom functionCallHandler, promptfoo forwards the model-emitted tool name and arguments to that handler. toolCallTimeout bounds each handler invocation, and maxToolIterations stops runaway tool-follow-up loops within a single turn. If the handler performs side effects, validate the function name and parse or validate the arguments before acting on them. For deterministic eval checks, use an is-valid-openai-tools-call assertion when you need to enforce an exact schema match.

Complete Example

For a complete working example that demonstrates the Realtime API capabilities, see the OpenAI Realtime API example or initialize it with:

npx promptfoo@latest init --example openai-realtime

This example includes:

• Basic single-turn interactions with the Realtime API
• Multi-turn conversations with persistent context
• Conversation threading with separate conversation IDs
• A runnable JavaScript function-calling example with a custom functionCallHandler
• JavaScript prompt function for properly formatting messages
• Detailed documentation on handling content types correctly

Input and Message Format

When using the Realtime API with promptfoo, you can specify the prompt in JSON format:

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "{{question}}"
      }
    ]
  }
]

Promptfoo preserves the native Realtime user-content items accepted by OpenAI, including input_text, input_audio, and input_image:

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "Describe these inputs."
      },
      {
        "type": "input_audio",
        "audio": "<base64-encoded audio>"
      },
      {
        "type": "input_image",
        "image_url": "data:image/jpeg;base64,..."
      }
    ]
  }
]

Use input_image only with Realtime models that support image input, such as the current gpt-realtime* family.

Multi-Turn Conversations

The Realtime API supports multi-turn conversations with persistent context. For implementation details and examples, see the OpenAI Realtime example, which demonstrates both single-turn interactions and conversation threading using the conversationId metadata property.

Important: When implementing multi-turn conversations, use type: "input_text" for user inputs and type: "text" for assistant responses.

Responses API

OpenAI's Responses API is the most advanced interface for generating model responses, supporting text and image inputs, function calling, and conversation state. It provides access to OpenAI's full suite of features including reasoning models like o1, o3, and o4 series.

Supported Responses Models

The Responses API supports a wide range of models, including:

• gpt-5.6 - Alias for GPT-5.6 Sol ($5/$30 per 1M tokens)
• gpt-5.6-sol - GPT-5.6 flagship model ($5/$30 per 1M tokens)
• gpt-5.6-terra - GPT-5.6 balanced model ($2.50/$15 per 1M tokens)
• gpt-5.6-luna - GPT-5.6 efficient model ($1/$6 per 1M tokens)
• gpt-5.5 - GPT-5.5 model ($5/$30 per 1M tokens)
• gpt-5.5-2026-04-23 - Dated snapshot of gpt-5.5
• gpt-5.5-pro - Premium GPT-5.5 model ($30/$180 per 1M tokens)
• gpt-5.5-pro-2026-04-23 - Dated snapshot of gpt-5.5-pro
• gpt-5.4 - GPT-5.4 model ($2.50/$15 per 1M tokens)
• gpt-5.4-2026-03-05 - Dated snapshot of gpt-5.4
• gpt-5.4-mini - Smaller GPT-5.4 model ($0.75/$4.50 per 1M tokens)
• gpt-5.4-mini-2026-03-17 - Dated snapshot of gpt-5.4-mini
• gpt-5.4-nano - Lowest-cost GPT-5.4 model ($0.20/$1.25 per 1M tokens)
• gpt-5.4-nano-2026-03-17 - Dated snapshot of gpt-5.4-nano
• gpt-5.4-pro - Premium GPT-5.4 model ($30/$180 per 1M tokens)
• gpt-5.4-pro-2026-03-05 - Dated snapshot of gpt-5.4-pro
• gpt-5 - Earlier GPT-5 family model
• gpt-5-chat - GPT-5 chat alias
• gpt-5.1 - GPT-5.1 base model
• gpt-5.1-chat-latest - GPT-5.1 chat alias
• gpt-5.3-chat-latest - GPT-5.3 chat alias
• gpt-5.2-chat-latest - GPT-5.2 chat-optimized alias
• gpt-5.2-codex - GPT-5.2 coding variant
• gpt-5.2-pro - Premium GPT-5.2 model with highest reasoning capability ($21/$168 per 1M tokens)
• o1 - Powerful reasoning model
• o1-mini - Smaller, more affordable reasoning model
• o1-pro - Enhanced reasoning model with more compute
• o3-pro - Highest-tier reasoning model
• o3 - General-purpose reasoning model
• o3-mini - Smaller, more affordable reasoning model
• o4-mini - Fast, cost-effective reasoning model
• gpt-5-codex - GPT-5 based coding model optimized for code generation
• gpt-5-codex-mini - Responses replacement for the retired codex-mini-latest model
• gpt-5-pro - Premium GPT-5 model with highest reasoning capability ($15/$120 per 1M tokens)

The Pro and Codex variants of GPT-5, GPT-5.1, GPT-5.2, and GPT-5.3, along with o1-pro, o3-pro, and computer-use-preview, are Responses-only. Promptfoo auto-routes their bare IDs to Responses. Several older Codex and computer-use snapshots have published July 23, 2026 shutdown dates; check OpenAI's deprecation schedule before adding new eval coverage. codex-mini-latest has already been shut down; OpenAI recommends gpt-5-codex-mini as its Responses API replacement. gpt-5-pro and its dated snapshot require reasoning.effort: high, while gpt-5.2-pro and its snapshot support medium, high, and xhigh.

Using the Responses API

To use the OpenAI Responses API, use the provider format openai:responses:<model name>:

providers:
  - id: openai:responses:gpt-5
    config:
      max_output_tokens: 500
      instructions: 'You are a helpful, creative AI assistant.'

Responses-specific Configuration Options

The Responses API configuration supports these parameters in addition to standard OpenAI parameters:

MCP (Model Context Protocol) Support

The Responses API supports OpenAI's MCP integration, allowing models to use remote MCP servers to perform tasks. MCP tools enable access to external services and APIs through a standardized protocol.

Basic MCP Configuration

To use MCP tools with the Responses API, add them to the tools array:

providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          require_approval: never

MCP Tool Configuration Options

Authentication with MCP Servers

Most MCP servers require authentication. Use the headers parameter to provide API keys or tokens:

providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: stripe
          server_url: https://mcp.stripe.com
          headers:
            Authorization: 'Bearer {{ env.STRIPE_API_KEY }}'
          require_approval: never

Filtering MCP Tools

To limit which tools are available from an MCP server, use the allowed_tools parameter:

providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          allowed_tools: ['ask_question']
          require_approval: never

Approval Settings

By default, OpenAI requires approval before sharing data with MCP servers. You can configure approval settings:

# Never require approval for all tools
providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          require_approval: never

  # Never require approval for specific tools only
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          require_approval:
            never:
              tool_names: ['ask_question', 'read_wiki_structure']

Complete MCP Example

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'What are the transport protocols supported in the MCP specification for {{repo}}?'

providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          require_approval: never
          allowed_tools: ['ask_question']

tests:
  - vars:
      repo: modelcontextprotocol/modelcontextprotocol
    assert:
      - type: contains
        value: 'transport protocols'

For a complete working example, see the OpenAI MCP example or initialize it with:

npx promptfoo@latest init --example openai-mcp

Reasoning Models

When using reasoning models like o1, o1-pro, o3, o3-pro, o3-mini, or o4-mini, you can control the reasoning effort:

providers:
  - id: openai:responses:o3
    config:
      reasoning:
        effort: 'medium' # Can be "low", "medium", or "high"
      max_output_tokens: 1000

Reasoning models "think before they answer," generating internal reasoning that isn't visible in the output but counts toward token usage and billing.

o3 and o4-mini Models

OpenAI offers advanced reasoning models in the o-series:

o3 and o4-mini

These reasoning models provide different performance and efficiency profiles:

• o3: Powerful reasoning model, optimized for complex mathematical, scientific, and coding tasks
• o4-mini: Efficient reasoning model with strong performance in coding and visual tasks at lower cost

Both models feature:

• Large context window (200,000 tokens)
• High maximum output tokens (100,000 tokens)

For current specifications and pricing information, refer to OpenAI API pricing.

Example configuration:

providers:
  - id: openai:responses:o3
    config:
      reasoning:
        effort: 'high'
      max_output_tokens: 2000

  - id: openai:responses:o4-mini
    config:
      reasoning:
        effort: 'medium'
      max_output_tokens: 1000

Deep Research Models (Responses API Only)

Deep research models (o3-deep-research, o4-mini-deep-research) are specialized reasoning models designed for complex research tasks that require web search capabilities.

Available models:

• o3-deep-research - Most powerful deep research model ($10/1M input, $40/1M output)
• o3-deep-research-2025-06-26 - Snapshot version
• o4-mini-deep-research - Faster, more affordable ($2/1M input, $8/1M output)
• o4-mini-deep-research-2025-06-26 - Snapshot version

OpenAI has scheduled these deep research aliases and snapshots to shut down on July 23, 2026 and recommends gpt-5.5-pro for new integrations. Retain them only for existing eval coverage.

All deep research models:

• Require at least one data source: web_search, web_search_preview, file_search with vector_store_ids, or an MCP tool
• Support 200,000 token context window
• Support up to 100,000 output tokens
• May take 2-10 minutes to complete research tasks
• Use significant tokens for reasoning before generating output

Example configuration:

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      max_output_tokens: 50000 # High limit recommended
      tools:
        - type: web_search_preview # Data source

Advanced Configuration

providers:
  - id: openai:responses:o3-deep-research
    config:
      max_output_tokens: 100000
      max_tool_calls: 50 # Limit searches to control cost/latency
      background: true # Recommended for long-running tasks
      store: true # Store conversation for 30 days
      tools:
        - type: web_search_preview # Data source
        - type: code_interpreter # Optional: For data analysis
          container:
            type: auto
        - type: mcp # Optional: Connect to private data
          server_label: mycompany_data
          server_url: https://api.mycompany.com/mcp
          require_approval: never # Must be 'never' for deep research

Response Format

Deep research models return specialized output items:

• web_search_call: Web search actions (search, open_page, find_in_page)
• code_interpreter_call: Code execution for analysis
• message: Final answer with inline citations and annotations

Example response structure:

{
  "output": [
    {
      "type": "web_search_call",
      "action": {
        "type": "search",
        "query": "latest AI research papers 2025"
      }
    },
    {
      "type": "message",
      "content": [
        {
          "type": "output_text",
          "text": "Based on my research...",
          "annotations": [
            {
              "url": "https://arxiv.org/...",
              "title": "Paper Title",
              "start_index": 123,
              "end_index": 145
            }
          ]
        }
      ]
    }
  ]
}

Best Practices

1. Use Background Mode: For production, always use background: true to handle long response times
2. Set High Token Limits: Use max_output_tokens: 50000 or higher
3. Configure Timeouts: Set PROMPTFOO_EVAL_TIMEOUT_MS=600000 for 10-minute timeouts
4. Control Costs: Use max_tool_calls to limit the number of searches
5. Enhance Prompts: Consider using a faster model to clarify/rewrite prompts before deep research

Timeout Configuration

Deep research models, GPT-5 pro variants, and any Responses request with background: true automatically use appropriate timeouts:

• If PROMPTFOO_EVAL_TIMEOUT_MS is set, it bounds the full API call, including background creation and polling
• Otherwise, these long-running requests default to a 10-minute timeout (600,000ms)
• Regular foreground requests continue to use the standard 5-minute timeout
• Background requests honor the overall creation and polling deadline. Non-cached requests cancel upstream work when an eval stops, including jobs accepted immediately before a delayed creation response arrives. Queued jobs in the shared persistent cache can be resumed by another eval process and are not cancelled when one local subscriber stops; use --no-cache for exclusive cancellation.
• Authenticated background requests are persisted only when an explicit, non-secret project or tenant header such as OpenAI-Project or X-Tenant-Id isolates the request. OpenAI-Organization alone does not isolate project API keys. When multiple eval processes resume the same persisted job, its completed usage is attributed exactly once. Running promptfoo cache clear also clears the associated billing-claim markers.
• Background streams preserve stream: true; promptfoo captures the response ID and cancels upstream work when an eval stops, including IDs that arrive shortly after the eval timeout

Example:

# Set a custom timeout for all evaluations
export PROMPTFOO_EVAL_TIMEOUT_MS=900000  # 15 minutes

# Or set the default API timeout (affects all providers)
export REQUEST_TIMEOUT_MS=600000  # 10 minutes

:::tip Deep research models require high max_output_tokens values (50,000+) and long timeouts. Set PROMPTFOO_EVAL_TIMEOUT_MS=600000 for 10-minute timeouts. :::

:::warning Deep research models require at least one supported data source. The provider will return an error if none of web_search, web_search_preview, file_search with vector_store_ids, or a remote MCP tool is configured. :::

GPT-5 Pro Timeout Configuration

gpt-5-pro, gpt-5.2-pro, gpt-5.4-pro, and gpt-5.5-pro are long-running models that often require extended timeouts due to advanced reasoning. Like deep research models, these variants automatically receive a 10-minute timeout (600,000ms) instead of the standard 5-minute timeout.

Automatic timeout behavior:

• GPT-5 pro variants automatically get a 10-minute timeout (600,000ms) - no configuration needed
• If you need longer, set PROMPTFOO_EVAL_TIMEOUT_MS (e.g., 900000 for 15 minutes)
• REQUEST_TIMEOUT_MS is ignored for GPT-5 pro variants (the automatic timeout takes precedence)

Most users won't need any timeout configuration - the automatic 10-minute timeout is sufficient for most GPT-5 pro requests.

If you experience timeouts, configure this:

# Only if you need more than the automatic 10 minutes
export PROMPTFOO_EVAL_TIMEOUT_MS=1200000   # 20 minutes

# For infrastructure reliability (recommended)
export PROMPTFOO_RETRY_5XX=true            # Retry 502 Bad Gateway errors
export PROMPTFOO_REQUEST_BACKOFF_MS=10000  # Longer retry backoff

# Reduce concurrency to avoid rate limits
promptfoo eval --max-concurrency 2

Common GPT-5 pro errors and solutions:

If you encounter errors with GPT-5 pro models:

1. Request timed out - If a GPT-5 pro model needs more than the automatic 10 minutes, set PROMPTFOO_EVAL_TIMEOUT_MS=1200000 (20 minutes)
2. 502 Bad Gateway - Enable PROMPTFOO_RETRY_5XX=true to retry Cloudflare/OpenAI infrastructure timeouts
3. getaddrinfo ENOTFOUND - Transient DNS errors; reduce concurrency with --max-concurrency 2
4. Upstream connection errors - OpenAI load balancer issues; increase backoff with PROMPTFOO_REQUEST_BACKOFF_MS=10000

:::tip GPT-5 pro models automatically get a 10-minute timeout. If you see infrastructure errors (502, DNS failures), enable PROMPTFOO_RETRY_5XX=true and reduce concurrency. :::

Sending Images in Prompts

The Responses API supports structured prompts with text, image, and file inputs. Example:

[
  {
    "type": "message",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "Describe what you see in this image about {{topic}}."
      },
      {
        "type": "input_image",
        "image_url": "{{image_url}}"
      }
    ]
  }
]

File inputs can use the same structured prompt format with type: "input_file". Set detail: "high" when you need higher-quality file rendering; otherwise OpenAI defaults to auto.

[
  {
    "type": "message",
    "role": "user",
    "content": [
      {
        "type": "input_text",
        "text": "Summarize the attached contract."
      },
      {
        "type": "input_file",
        "file_id": "file_abc123",
        "detail": "high"
      }
    ]
  }
]

Prompt Caching and Included Tool Results

Use prompt_cache_key for stable repeated prefixes and prompt_cache_retention: 24h when you want extended prompt caching on older models. GPT-5.5 and GPT-5.5 Pro Responses require extended retention, so prompt_cache_retention: in_memory will fail there. GPT-5.6 and later deprecate prompt_cache_retention; use prompt_cache_options.ttl instead.

GPT-5.6 also supports prompt_cache_options. The implicit mode below keeps automatic breakpoint placement for ordinary prompts. When the API reports cache_write_tokens, Promptfoo prices those writes at 1.25 times the active input rate.

providers:
  - id: openai:responses:gpt-5.6
    config:
      prompt_cache_key: shared-prefix
      prompt_cache_options:
        mode: implicit
        ttl: 30m

If you select explicit mode, add prompt_cache_breakpoint: { mode: explicit } to a supported structured content block. Explicit mode without a breakpoint disables prompt caching.

The include option requests extra structured payloads in the raw Responses object. For example, web_search_call.results returns search results when you need to inspect them in assertions or downstream tooling:

providers:
  - id: openai:responses:gpt-5.5
    config:
      prompt_cache_key: repeated-policy-prefix
      prompt_cache_retention: 24h
      include:
        - web_search_call.results
        - reasoning.encrypted_content

Function Calling

The Responses API supports tool and function calling, similar to the Chat API:

providers:
  - id: openai:responses:gpt-5
    config:
      tools:
        - type: function
          name: get_weather
          description: Get the current weather for a location
          parameters:
            type: object
            properties:
              location:
                type: string
                description: The city and state, e.g. San Francisco, CA
            required: ['location']
      tool_choice: 'auto'

Using with Azure

The Responses API can also be used with Azure OpenAI endpoints by configuring the apiHost:

providers:
  - id: openai:responses:gpt-4.1
    config:
      apiHost: 'your-resource.openai.azure.com'
      apiKey: '{{ env.AZURE_API_KEY }}' # or set OPENAI_API_KEY env var
      temperature: 0.7
      instructions: 'You are a helpful assistant.'
      response_format: file://./response-schema.json

Legacy apiHost, newer apiBaseUrl, and OpenAI endpoint environment variable Azure configurations all support the same Responses reasoning and verbosity options.

For comprehensive Azure Responses API documentation, see the Azure provider documentation.

Complete Example

For a complete working example, see the OpenAI Responses API example or initialize it with:

npx promptfoo@latest init --example openai-responses

Troubleshooting

OpenAI rate limits

Promptfoo automatically handles OpenAI rate limits with retry and adaptive concurrency. See Rate Limits for details.

If you need manual control, you can:

1. Reduce concurrency with --max-concurrency 1 in the CLI or evaluateOptions.maxConcurrency in config
2. Add fixed delays with --delay 3000 (milliseconds) or evaluateOptions.delay in config
3. Adjust backoff with PROMPTFOO_REQUEST_BACKOFF_MS environment variable (default: 5000ms)

OpenAI flakiness

To retry HTTP requests that are Internal Server errors, set the PROMPTFOO_RETRY_5XX environment variable to 1.

Agentic Providers

OpenAI offers several agentic providers for different use cases:

Agents SDK

Test multi-turn agentic workflows with the OpenAI Agents provider. This provider supports the @openai/agents SDK with tools, handoffs, sessions, sandbox agents, and tracing. For the Python openai-agents SDK, use the OpenAI Agents Python SDK guide.

providers:
  - id: openai:agents:my-agent
    config:
      agent: file://./agents/support-agent.ts
      tools: file://./tools/support-tools.ts
      maxTurns: 10
      modelSettings:
        retry:
          maxRetries: 2
          policy: providerSuggested

See the OpenAI Agents documentation for full configuration options, retry policies, and examples.

Codex SDK

For agentic coding tasks with working directory access and structured JSON output, use the OpenAI Codex SDK provider. This provider supports the GPT-5.6 family, GPT-5.5, GPT-5.4, and Codex-optimized GPT-5 models for code generation. You can select a model inline with openai:codex:gpt-5.6-sol or via config.model when you need additional options:

Promptfoo preserves SDK-reported input, output, cached input, and reasoning output tokens in tokenUsage when Codex returns them.

providers:
  - id: openai:codex-sdk
    config:
      model: gpt-5.6-sol
      working_dir: ./src
      output_schema:
        type: object
        properties:
          code: { type: string }
          explanation: { type: string }

See the OpenAI Codex SDK documentation for thread management, structured output, and Git-aware operations.

Codex App Server

For app-server protocol evals, use the OpenAI Codex App Server provider. It starts codex app-server as a local child process and is intended for rich-client behavior such as streamed app-server items, approval requests, MCP elicitations, skill/plugin/app connector events, and thread lifecycle metadata. It does not attach to an already-running Codex Desktop app process.