agent-eval/README.md
Runs coding agents (Claude Code and Codex) against fixture projects in sandboxes and asserts that they follow the Storybook workflows this repo ships — writing stories, previewing or reviewing them, and running story tests through the MCP server or the plugin skills.
Install dependencies:
yarn install
Configure environment variables:
cp .env.example .env.local
Edit .env.local and add your API keys (see comments in .env.example for options):
ANTHROPIC_API_KEY is required for the Claude Code experiments and for failure classification, both of which use the direct Anthropic API. OPENAI_API_KEY is required for the Codex experiments, which use the direct Codex API.sandbox: 'auto', which uses Vercel Sandbox when access-token credentials (VERCEL_PROJECT_ID, VERCEL_TEAM_ID, and VERCEL_TOKEN) are present and falls back to local Docker otherwise. Set sandbox: 'docker' to force Docker-only experiments.Run the commands below from the repository root with
yarn workspace agent-eval run <script>.
See what will run without making API calls:
yarn workspace agent-eval run eval:dry
Run all configured experiments:
yarn workspace agent-eval run eval
Run a single experiment:
yarn workspace agent-eval exec agent-eval cc-mcp-opus-high
Pull requests with the ci:eval label run all experiments in CI. The
ci:eval/ci:extra-* labels are applied by humans only — labeled runs are
expensive and re-trigger on every subsequent push, so an AI agent must never
add them to a PR (nor start workflow_dispatch eval runs). Agents validate
their changes locally instead: only the specific evals affected by the change
(or the eval being fixed), one experiment at a time, via EVAL_ONLY — never a
full line, never multiple experiments in parallel.
By default only the first core eval (801-create-component-no-launch-config)
runs. Set EVAL_EXTRA_EVALS=1 to run the full hand-crafted line — the 8xx
workflow evals on every experiment plus the lifecycle 82x evals
(storybook-init/storybook-upgrade scenarios) on the plugin experiments —
or EVAL_ONLY=<name>[,<name>] to debug specific evals one at a time:
EVAL_EXTRA_EVALS=1 yarn workspace agent-eval run eval
EVAL_ONLY=803-edit-component yarn workspace agent-eval run eval
Before a local run, rebuild the local @storybook/addon-mcp/@storybook/mcp
builds the sandboxes inject (yarn nx run-many -t compile --projects mcp,addon-mcp
from the repository root). A stale dist importing since-renamed core exports
crashes the sandbox Storybook at preset load, which surfaces as the readiness
timeout below rather than a build error.
A full EVAL_EXTRA_EVALS=1 run (12 workflow evals × 4 experiments + 3
lifecycle evals × 2 plugin experiments) costs roughly $30–45 in agent
tokens at current per-run averages ($0.30–0.80 per workflow eval, $1–2 per
lifecycle eval). The budget guardrail is $75 per full run — check the
usage metadata in the results playground before growing the eval set past it
(see storybookjs/mcp#324).
The 9xx evals are a trimmed MCP-only set for shapes the 8xx line does not
cover (async mocks, story drift, tool params, preview-by-path/id, vitest CLI).
They never run on the default next matrix; under EVAL_STORYBOOK_LATEST=1
they become the active line (default smoke: 908-run-story-tests). See
lib/experiment.ts. Twins of 8xx scenarios were removed.
Experiments named <agent>-<integration>-<model>-<effort> pin their model and
effort explicitly. Non-default model tiers (currently cc-plugin-sonnet-medium and cc-mcp-sonnet-medium)
run zero evals unless EVAL_EXTRA_MODELS=1 is set, so labeled CI runs only pay
for the default-model experiments:
EVAL_EXTRA_MODELS=1 yarn workspace agent-eval exec agent-eval cc-plugin-sonnet-medium
Sandbox setup resolves the Storybook npm dist-tag at run time and pins the
exact version it finds into the sandbox package.json, so each result snapshot
records which version the run used. By default it pins the next tag and keeps
the local @storybook/addon-mcp/@storybook/mcp builds from this checkout.
Set EVAL_STORYBOOK_LATEST=1 to pin the latest tag instead — including the
published @storybook/addon-mcp and @storybook/mcp in place of the local
builds — to check whether a behavior change (e.g. in the documentation tooling)
regressed since the last stable release:
EVAL_STORYBOOK_LATEST=1 yarn workspace agent-eval run eval
Review mode follows the integration. The plugin experiments always run — and
assert — the review workflow (review-create published, review section in the
final response), because review is on by default for the storybook ai CLI
channel the plugins use. The MCP experiments run review-off by default
(stories-preview links, no review-create), matching direct MCP clients where
the experimentalReview feature flag is opt-in. Set EVAL_REVIEW=1 to enable
the flag in every sandbox Storybook and flip the MCP assertions to the review
workflow too:
EVAL_REVIEW=1 yarn workspace agent-eval run eval
In CI, the ci:extra-evals, ci:extra-models, ci:storybook-latest, and
ci:review PR labels set the matching flag on labeled ci:eval runs, and
manual workflow_dispatch runs of the Agent eval workflow can enable them
through the extra_evals, extra_models, storybook_latest, and review
inputs, or target specific evals through the eval_only input. All of these
are human-triggered spend decisions; agents never apply the labels or dispatch
the workflow.
CI uses Vercel Sandbox through access-token credentials (VERCEL_PROJECT_ID,
VERCEL_TEAM_ID, and VERCEL_TOKEN). Do not store a static
VERCEL_OIDC_TOKEN in GitHub secrets; development OIDC tokens expire and
Vercel-issued OIDC is only refreshed automatically inside Vercel-managed
runtime/build contexts.
Configured experiments (Claude Code experiments use the direct Anthropic API
via ANTHROPIC_API_KEY; Codex experiments use the direct Codex API via
OPENAI_API_KEY):
cc-mcp-opus-high: Claude Code (Opus at high effort) with project-local Storybook MCP config in .mcp.json.cc-plugin-opus-high: Claude Code (Opus at high effort) with Storybook plugin skills copied to .claude/skills.codex-mcp-gpt-5.5-medium: Codex (gpt-5.5 at medium reasoning effort) with project-local Storybook MCP config in .codex/config.toml.codex-plugin-gpt-5.5-medium: Codex (gpt-5.5 at medium reasoning effort) with Storybook plugin skills copied to .agents/skills.cc-mcp-sonnet-medium / cc-plugin-sonnet-medium: Claude Code (Sonnet at medium effort) variants; they run zero evals unless EVAL_EXTRA_MODELS=1 is set.Accepted eval failures are documented as a code comment directly above the
relaxed assertion in the affected EVAL.ts. The comment is self-contained:
the observed behavior, the evidence (CI run id and date), and the condition
for re-enabling. See the gates in evals/807-docs-request/EVAL.ts and
evals/808-shared-infra-fallback/EVAL.ts for the expected shape.
Fixtures can opt into a shared starter project with package metadata:
{
"evals": {
"template": "reshaped-storybook"
}
}
Templates live in agent-eval/templates/<template-name> and are copied into the
sandbox during setup before the agent runs. They intentionally stay visible in
saved result project snapshots so eval runs are easy to inspect.
Three templates exist today:
reshaped-storybook: the design-system shape — Reshaped components, full
Storybook (next) with the local addon builds, MSW, and the vitest story
test setup.vite-app: a minimal React + Vite app with no Storybook at all. The
lifecycle fixtures use it directly (820 init) or layer an old Storybook on
top (821/822 upgrades and 823 setup-on-outdated, which also set
evals.pinStorybook: false so the harness keeps their intentionally
outdated versions); 812 layers a full Storybook next setup with zero
stories on top.monorepo: an npm-workspaces repo where the runnable Storybook lives in the
packages/ui leaf, so evals can cover agents working inside a workspace
package. Storybook pinning and the local file: build detection cover
workspace package.json files too.This keeps prompt variants small: each variant keeps its own PROMPT.md,
EVAL.ts, and metadata package.json, while shared app files stay in the
template.
Templates can use local built Storybook MCP packages with npm file:
dependencies, for example file:./local-packages/addon-mcp. The setup step
copies code/addons/mcp/dist and code/lib/mcp/dist from this checkout into
the sandbox before the sandbox runs npm install. CI builds those packages
before running evals; run yarn nx run-many -t compile --projects mcp,addon-mcp
locally after changing those packages.
The MCP experiments configure each agent through its project-local MCP file:
Claude Code gets .mcp.json, and Codex gets .codex/config.toml. The plugin
experiments do not write MCP config; they copy the Storybook plugin skills into
the agent's project skill directory instead. The template is responsible for
starting Storybook before the agent runs; reshaped-storybook does this from
postinstall so it runs after sandbox dependencies are installed.
Codex experiments use the direct codex agent with OPENAI_API_KEY. The
codex-mcp experiment cannot use vercel-ai-gateway/codex until the Gateway
path handles Codex's Responses namespace tool shape reliably. See
https://github.com/openai/codex/issues/26234.
yarn workspace agent-eval run playground
Open http://localhost:3000 to browse results.
Pull the eval results produced by recent CI runs into the local
agent-eval/results directory, so they can be browsed in the local playground
and inspected by analysis tooling:
yarn workspace agent-eval run results:download # latest 20 agent-eval-results artifacts
yarn workspace agent-eval run results:download 5 # or any count between 1 and 100
Requires an authenticated GitHub CLI (gh auth login) and a tar binary
(preinstalled on macOS and Linux). Result snapshots are
keyed by experiment name and run timestamp, so artifacts from multiple CI runs
merge into agent-eval/results without colliding, and re-running the command
is idempotent. Each artifact is roughly 20–40 MB extracted.
The Agent eval GitHub Actions workflow deploys the playground to Vercel
project storybook-evals after eval results have been written to
agent-eval/results.
ci:eval label create
preview deployments.main branches create preview deployments.main create production deployments.The workflow deploys from the same runner that produced agent-eval/results,
so failed evals can still publish a playground with partial results. The final
workflow status still fails when the eval, build, or deploy step fails.
The workflow links the Vercel project at runtime instead of committing
.vercel/project.json. It uses the same Vercel access token for the Sandbox
evals and the Vercel CLI preview deployment, but those are separate steps:
Sandbox auth happens in yarn workspace agent-eval run eval, while the preview playground deployment
runs vercel link, vercel pull, vercel build, and
vercel deploy --prebuilt.
Configure these GitHub secrets before enabling the workflow:
VERCEL_TOKEN: Vercel access token with Sandbox and deploy access to the Storybook team.VERCEL_TEAM_ID: Vercel team ID or slug for the Storybook account.VERCEL_PROJECT_ID: Vercel project ID used by Vercel Sandbox access-token auth.The thin app wrapper in agent-eval/app re-exports routes from
@vercel/agent-eval-playground so Next.js can discover them from this package.
Run yarn workspace agent-eval run playground:check-routes after upgrading the playground package.