OpenClaw

OpenClaw is a personal AI assistant framework that enables agentic evaluations with configurable reasoning and session management.

Prerequisites

1. Install OpenClaw:

npm install -g openclaw@latest

2. Run the onboarding wizard:

openclaw onboard

3. Enable the HTTP API in ~/.openclaw/openclaw.json if you want Chat or Responses. These HTTP endpoints are disabled by default upstream:

{
  "gateway": {
    "http": {
      "endpoints": {
        "chatCompletions": {
          "enabled": true
        },
        "responses": {
          "enabled": true
        }
      }
    }
  }
}

4. Start the gateway:

openclaw gateway

Or restart if already running:

openclaw gateway restart

Provider Types

OpenClaw exposes five provider types, each targeting a different gateway API surface:

Chat (default)

Uses the OpenAI-compatible chat completions endpoint. This is the default when no keyword is specified. Requires gateway.http.endpoints.chatCompletions.enabled=true.

• openclaw - Uses the default agent
• openclaw:main - Explicitly targets the main agent
• openclaw:<agent-id> - Targets a specific agent by ID

Promptfoo sends OpenClaw's current slash-style model id (openclaw/<agent-id>) to the gateway while keeping the openclaw:<agent-id> promptfoo provider syntax for compatibility.

Responses

Uses the OpenResponses-compatible /v1/responses endpoint. This endpoint is also disabled by default and requires enabling in gateway config:

{
  "gateway": {
    "http": {
      "endpoints": {
        "responses": { "enabled": true }
      }
    }
  }
}

• openclaw:responses - Default agent via Responses API
• openclaw:responses:main - Explicit agent ID
• openclaw:responses:<agent-id> - Custom agent

Embeddings

Uses the OpenAI-compatible /v1/embeddings endpoint. The model field selects the OpenClaw agent target, and config.backend_model can override the backend embedding model with the x-openclaw-model header.

• openclaw:embedding - Default agent via Embeddings API
• openclaw:embedding:<agent-id> - Custom agent by ID
• openclaw:embeddings:<agent-id> - Plural alias (same behavior)

WebSocket Agent

Uses the native OpenClaw WebSocket RPC protocol for full agent streaming. Connects directly to the gateway's WS port without requiring HTTP endpoint enablement. Promptfoo includes a stable device identity, signs the gateway connect.challenge nonce, persists issued hello-ok.auth.deviceToken values, and retries once with a cached device token when the gateway reports an AUTH_TOKEN_MISMATCH.

• openclaw:agent - Default agent via WS
• openclaw:agent:main - Explicit agent ID
• openclaw:agent:<agent-id> - Custom agent

Tool Invoke

Invokes a specific tool directly via POST /tools/invoke. Useful for testing stable built-in tools in isolation. The prompt is parsed as JSON for tool arguments.

:::note If the tool isn't allowlisted by OpenClaw policy, the gateway returns a 404 error. Start with a stable built-in tool such as sessions_list or session_status. Tools like bash may be renamed, aliased, or blocked by policy depending on your OpenClaw setup. :::

:::tip POST /tools/invoke also has an upstream HTTP deny list by default. Expect 404s for tools such as sessions_spawn, sessions_send, cron, gateway, and whatsapp_login unless your OpenClaw policy explicitly changes that behavior. :::

• openclaw:tools:sessions_list - Invoke the sessions_list tool
• openclaw:tools:session_status - Invoke the session_status tool

Configuration

Auto-Detection

The provider automatically detects the gateway URL and bearer auth secret from the active OpenClaw config (OPENCLAW_CONFIG_PATH when set, otherwise ~/.openclaw/openclaw.json). This includes:

• local bind/port resolution
• OPENCLAW_GATEWAY_PORT as a local port override
• gateway.tls.enabled for https:// / wss://
• gateway.mode=remote via gateway.remote.url

providers:
  - openclaw:main

Explicit Configuration

Override auto-detection with explicit config:

providers:
  - id: openclaw:main
    config:
      gateway_url: http://127.0.0.1:18789
      auth_token: your-token-here
      # Use auth_password instead when gateway.auth.mode=password
      session_key: custom-session
      # Optional backend model override, sent as x-openclaw-model:
      backend_model: openai/gpt-5.4

Environment Variables

Set configuration via environment variables:

export OPENCLAW_CONFIG_PATH=~/.openclaw/openclaw.json  # optional
export OPENCLAW_GATEWAY_URL=http://127.0.0.1:18789
# Or override only the local auto-detected port:
# export OPENCLAW_GATEWAY_PORT=18789
export OPENCLAW_GATEWAY_TOKEN=your-token-here
# Or, if your gateway uses password auth:
# export OPENCLAW_GATEWAY_PASSWORD=your-password-here

providers:
  - openclaw:main

Config Options

Examples

Basic Usage

prompts:
  - 'What is the capital of {{country}}?'

providers:
  - openclaw:main

tests:
  - vars:
      country: France
    assert:
      - type: contains
        value: Paris

With Custom Thinking Level (WS Agent)

thinking_level is only supported by the WebSocket Agent provider. Valid values are off, minimal, low, medium, high, xhigh, and adaptive, though model support still depends on the upstream provider/model combination.

prompts:
  - 'Analyze the pros and cons of {{topic}}'

providers:
  - id: openclaw:agent:main
    config:
      session_key: promptfoo-eval
      thinking_level: adaptive
      timeoutMs: 60000

tests:
  - vars:
      topic: renewable energy

Using Responses API

prompts:
  - 'Summarize: {{text}}'

providers:
  - openclaw:responses:main

tests:
  - vars:
      text: The quick brown fox jumps over the lazy dog.

Using Embeddings

prompts:
  - '{{text}}'

providers:
  - id: openclaw:embedding:main
    config:
      backend_model: openai/text-embedding-3-small

tests:
  - vars:
      text: Promptfoo routes this through OpenClaw.

Backend Model Override

Use backend_model when you want the selected OpenClaw agent to run a specific provider/model for this eval without changing the agent's normal default model.

providers:
  - id: openclaw:main
    config:
      backend_model: openai/gpt-5.4

For billing, OpenClaw's visible model remains the agent target (openclaw/<agent-id>). Promptfoo can estimate OpenAI token spend only when backend_model or model_override names the actual OpenAI backend model, such as openai/gpt-5.4 or gpt-5.4. If the backend model is selected only inside OpenClaw's own agent config, Promptfoo leaves cost unset because the gateway response does not expose enough information to price that request safely.

WebSocket Agent

Promptfoo uses an isolated session key per call unless you set session_key explicitly.

prompts:
  - '{{task}}'

providers:
  - id: openclaw:agent:main
    config:
      session_key: promptfoo-eval
      timeoutMs: 60000

tests:
  - vars:
      task: What files are in the current directory?

Tool Invoke

prompts:
  - '{}'

providers:
  - openclaw:tools:sessions_list

tests:
  - assert:
      - type: contains
        value: sessions

If a tool exposes sub-actions, add config.action:

prompts:
  - '{}'

providers:
  - id: openclaw:tools:sessions_list
    config:
      action: json

Troubleshooting

• 404 from openclaw:main or openclaw:responses:*: the HTTP endpoints are disabled by default. Enable gateway.http.endpoints.chatCompletions.enabled=true and, for Responses, gateway.http.endpoints.responses.enabled=true.
• 404 from openclaw:tools:*: the tool may be blocked by gateway.tools, the default HTTP deny list, or your selected tools.profile. Start with sessions_list or session_status.
• WS agent auth failures on password-mode gateways: use auth_password or OPENCLAW_GATEWAY_PASSWORD, not auth_token.
• WS DEVICE_AUTH_* errors usually mean an old or incompatible device identity/signature. Remove only the promptfoo OpenClaw device identity/cache files you configured, then pair again.
• If you use unusual proxying or a nonstandard gateway URL, set gateway_url explicitly instead of relying on auto-detection.

See Also

For a complete example, see examples/provider-openclaw.