ContextQMD
Back to Promptfoo

openai-chatkit (OpenAI ChatKit Workflow Evaluation)

examples/openai-chatkit/README.md

0.121.91.8 KB
Original Source

openai-chatkit (OpenAI ChatKit Workflow Evaluation)

Evaluate ChatKit workflows built with OpenAI's Agent Builder using browser automation.

Quick Start

bash
npx promptfoo@latest init --example openai-chatkit
cd openai-chatkit

Prerequisites

  1. Install Playwright browser:
bash
npx playwright install chromium

  1. Get your workflow ID from Agent Builder:

    • Open platform.openai.com > Agent Builder
    • Click Publish on your workflow
    • Copy the workflow ID (e.g., wf_68ffb83dbfc88190a38103c2bb9f421003f913035dbdb131)

  2. Update promptfooconfig.yaml with your workflow ID

Environment Variables

  • OPENAI_API_KEY - Your OpenAI API key

Run the Eval

bash
npx promptfoo@latest eval --max-concurrency 4

View results:

bash
npx promptfoo@latest view

Multi-Turn Conversations

For workflows that ask follow-up questions, enable stateful mode:

yaml
providers:
  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
    config:
      stateful: true

tests:
  - vars:
      message: 'I want to plan a party'
  - vars:
      message: 'For 20 people with a $500 budget'

Run with --max-concurrency 1 for stateful mode.

Using with Simulated User

Test multi-turn conversations automatically:

yaml
providers:
  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
    config:
      stateful: true

defaultTest:
  provider:
    id: 'promptfoo:simulated-user'
    config:
      maxTurns: 5

tests:
  - vars:
      instructions: |
        You are planning a birthday party.
        Answer questions naturally and provide details when asked.

Learn More