OpenAI Codex App Server

This provider starts codex app-server as a local child process and drives the Codex app-server JSON-RPC protocol from promptfoo. Use it when you need to eval the rich client surface of Codex: streamed agent items, approvals, skills, plugins, app connector events, command/file trajectories, and thread lifecycle metadata.

For CI and straightforward automation, prefer the OpenAI Codex SDK provider. The app-server protocol is experimental, broader than the SDK, and designed for rich product integrations.

Provider IDs

providers:
  - openai:codex-app-server
  - openai:codex-app-server:gpt-5.5
  - openai:codex-desktop
  - openai:codex-desktop:gpt-5.5

openai:codex-desktop is an alias for the same app-server protocol. Promptfoo starts its own codex app-server process; it does not attach to an already-running Codex Desktop app process.

Codex SDK vs App Server vs Desktop App

Keep this provider separate from the Codex SDK provider. They share Codex concepts, but they expose different runtime contracts.

Use this provider when the thing being tested depends on app-server-only behavior such as approval request payloads, streamed item notifications, app connector events, plugin/skill metadata, or thread lifecycle operations. Use the SDK provider when you only need final Codex output, thread reuse, structured output, and traced shell/MCP/search/file steps.

What Promptfoo Can and Can't Evaluate

Setup

Install the Codex CLI and sign in:

npm i -g @openai/codex
codex

You can also authenticate with an API key:

export OPENAI_API_KEY=your_api_key_here

Promptfoo also accepts CODEX_API_KEY or config.apiKey. For reproducible evals, prefer API-key-backed runs or set cli_env.CODEX_HOME to a fixture home directory that already contains the intended Codex login state.

Basic Usage

providers:
  - id: openai:codex-app-server:gpt-5.5
    config:
      sandbox_mode: read-only
      approval_policy: never

prompts:
  - 'Review this repository and summarize the highest-risk code paths.'

The provider returns Codex's final assistant text as output. It also records thread ids, turn ids, item counts, command/file/tool metadata, approval decisions, and token usage under metadata.codexAppServer.

Safety Defaults

The app-server protocol can expose shell, filesystem, config, plugin, MCP, and app connector surfaces. Promptfoo defaults to deterministic eval behavior:

Approval requests are answered without blocking:

Use accept, acceptForSession, permission grants, or MCP elicitation acceptance only in isolated workspaces where side effects are acceptable.

Configuration

The provider validates top-level provider config strictly. Prompt-level config is parsed more leniently because promptfoo merges generic test options into prompt.config; unrelated keys are ignored there, while invalid values for known Codex fields still return a row-level provider error.

Granular Approval Policy

providers:
  - id: openai:codex-app-server:gpt-5.5
    config:
      approval_policy:
        granular:
          sandbox_approval: true
          rules: true
          skill_approval: false
          request_permissions: true
          mcp_elicitations: true

Collaboration Mode

providers:
  - id: openai:codex-app-server:gpt-5.5
    config:
      collaboration_mode:
        mode: plan
        settings:
          model: gpt-5.5
          reasoning_effort: none
          developer_instructions: null

collaboration_mode is experimental and is sent on turn/start. App-server may let the selected mode override model, reasoning effort, or developer instructions for the turn.

Server Request Policy

Configure deterministic responses when you intentionally want app-server approval flows:

providers:
  - id: openai:codex-app-server:gpt-5.5
    config:
      sandbox_mode: workspace-write
      approval_policy: on-request
      server_request_policy:
        command_execution: decline
        file_change: decline
        user_input:
          severity: high
        mcp_elicitation:
          action: accept
          content:
            severity: low
          _meta:
            source: promptfoo
        permissions:
          scope: session
          permissions:
            network:
              enabled: true
            fileSystem:
              read:
                - /tmp/fixture
              write: null
        dynamic_tools:
          classify:
            success: true
            text: '{"label":"safe"}'

For command execution approvals, command_execution may also be an app-server decision object:

server_request_policy:
  command_execution:
    applyNetworkPolicyAmendment:
      network_policy_amendment:
        host: registry.npmjs.org
        action: allow

Legacy execCommandApproval and applyPatchApproval callbacks are also handled for older app-server versions. Advanced command decision objects are only supported on the modern item/commandExecution/requestApproval flow.

Structured Output

providers:
  - id: openai:codex-app-server:gpt-5.5
    config:
      sandbox_mode: read-only
      output_schema:
        type: object
        properties:
          summary:
            type: string
          risks:
            type: array
            items:
              type: string
        required: [summary, risks]
        additionalProperties: false

prompts:
  - 'Return a JSON review summary for this repo.'

tests:
  - assert:
      - type: is-json

The final app-server response is returned as a string. Use is-json or a JavaScript assertion to parse it.

Prompt Inputs

Plain text prompts work as usual. To include images, skills, or mentions, pass a JSON array:

[
  { "type": "text", "text": "$skill-creator Write a test plan for this provider." },
  { "type": "image", "url": "https://example.com/screenshot.png" },
  { "type": "local_image", "path": "/Users/me/screenshots/failure.png" },
  {
    "type": "skill",
    "name": "skill-creator",
    "path": "/Users/me/.codex/skills/skill-creator/SKILL.md"
  },
  {
    "type": "mention",
    "name": "workspace",
    "path": "app://connector/resource"
  }
]

Supported input item types are text, image, local_image, localImage, skill, and mention.

Metadata

The provider records app-server details for assertions and debugging:

providerResponse.metadata.codexAppServer.threadId;
providerResponse.metadata.codexAppServer.turnId;
providerResponse.metadata.codexAppServer.itemCounts;
providerResponse.metadata.codexAppServer.items;
providerResponse.metadata.codexAppServer.serverRequests;

Command output, tool arguments, and approval metadata are sanitized before they are placed in metadata or tracing attributes.

Tracing

Promptfoo wraps each provider call in a GenAI span. The app-server provider also creates item-level spans for completed command, file, MCP, dynamic tool, reasoning, search, and agent-message items.

Enable deeper app-server tracing by setting deep_tracing: true with Promptfoo's OpenTelemetry tracing enabled. Deep tracing starts a fresh app-server process for each row so the child process can receive the active trace context. Reusable app-server process and persistent thread pooling are disabled in this mode; explicit thread_id resumes are still serialized so parallel rows do not overlap turns on the same Codex thread.

Local Verification

Run from the repository root:

npm run local -- eval -c examples/openai-codex-app-server/promptfooconfig.yaml --no-cache

Use --env-file .env if your API key is stored there.

To validate the provider against your installed Codex CLI schema:

codex app-server generate-ts --out /tmp/codex-app-server-schema/ts
codex app-server generate-json-schema --out /tmp/codex-app-server-schema/json