Sequential Upgrade Benchmark — Operator Runbook

A practical reference for running the LLM cost-to-done benchmark end to end. Backend-agnostic, with MongoDB as the working example. Pair it with SpacetimeDB for the head-to-head.

What it measures: the total LLM cost (tokens + $), bug rate, and fix iterations to build the same chat app on each backend, graded against the same feature spec. Lower cost / fewer bugs / fewer iterations = easier to build on.

0. Backends, ports, layout (reference)

Backend	Vite (client)	API	Database	Title
`spacetime`	6173	(module on STDB :3000)	SpacetimeDB	"SpacetimeDB Chat"
`postgres`	6273	Express :6001	Postgres :6432	"PostgreSQL Chat"
`mongodb`	6373	Express :6001	Mongo :6437	"MongoDB Chat"

Parallel runs: --run-index N adds N to each port (6373+N, 6001+N, …) and uses a per-run database (chat-app_runN).

Run output lands here (gitignored, published separately to the results repo):

sequential-upgrade/sequential-upgrade-YYYYMMDD/
  <backend>/
    results/chat-app-<ts>/        # the generated app
      backend/ | server/  + client/
      .benchmark-backend          # marker used by fix/upgrade/grade detection
      level-1/ … level-11/        # snapshots taken before each upgrade
      BUG_REPORT.md               # you write this when grading finds bugs
      ITERATION_LOG.md            # fix history (appended each iteration)
      GRADING_RESULTS.md          # your per-feature scores
    telemetry/<run-id>/
      cost-summary.json | COST_REPORT.md | metadata.json
    inputs/                       # frozen prompt snapshot (reproducibility)
  BENCHMARK_REPORT.md             # generated by generate-report.mjs

Level → features graded (each feature scored 0–3):

Level	Features	Max	Level	Features	Max
1	1–4	12	7	1–10	30
2	1–5	15	8	1–11	33
3	1–6	18	9	1–12	36
4	1–7	21	10	1–13	39
5	1–8	24	11	1–14	42
6	1–9	27	12	1–15	45

1. One-time setup

Docker running. Brings up OTel collector + Postgres + Mongo:

bash

cd tools/llm-sequential-upgrade
docker compose -f docker-compose.otel.yaml up -d

SpacetimeDB (only if running the spacetime backend):
bash
```
spacetime start          # in its own terminal
```
Claude CLI on PATH (or npx @anthropic-ai/claude-code), and Node.js.
Run under git-bash on Windows (the scripts assume it; .gitattributes keeps them LF so they also work under WSL/CI).

2. Pre-run checklist (every session)

bash

cd tools/llm-sequential-upgrade
docker compose -f docker-compose.otel.yaml up -d        # idempotent
docker exec llm-sequential-upgrade-mongodb-1 mongosh --quiet --eval "db.runCommand({ping:1})"
# spacetime server ping local   # only for spacetime backend

All green → ready.

3. Pin the model (IMPORTANT for parity)

Pin the same model for every backend and every level. The published runs used Claude Sonnet 4.6. Two equivalent ways:

bash

./run.sh --model claude-sonnet-4-6 --level 1 --backend mongodb   # per-run flag
# or set it for the shell (also covers batch runs via run-loop/benchmark):
export ANTHROPIC_MODEL=claude-sonnet-4-6

--model overrides the env var. The chosen model is recorded in each run's metadata.json and printed in the run header — it's the single biggest comparability lever, so keep it identical across the whole comparison.

4. The core loop (per level, L1 → L12)

For each backend you're testing (mongodb, and spacetime if pairing):

4a. Generate L1 (or upgrade to level N)

bash

# Level 1 from scratch:
./run.sh --level 1 --backend mongodb

# Upgrade an existing app to the next level (default = incremental feature file;
# add --composed-prompt to use the full cumulative spec like the L1–L11 canon):
./run.sh --upgrade <app-dir> --level 2

Output ends with DEPLOY_COMPLETE and prints the app dir and COST_REPORT path. Backend is auto-detected for --upgrade/--fix via the .benchmark-backend marker, so you don't pass --backend again.

4b. Grade (manual — see §5)

Test every feature in the browser, score 0–3, write GRADING_RESULTS.md. If any feature fails, write BUG_REPORT.md (template in templates/).

4c. Fix (if bugs)

bash

./run.sh --fix <app-dir>

Reads BUG_REPORT.md, fixes, redeploys, appends to ITERATION_LOG.md. Re-grade. Repeat 4b–4c until all features pass (or you hit your iteration cap). Delete BUG_REPORT.md once everything passes — the harness keys --fix on its presence.

4d. Next level

Back to 4a with --upgrade <app-dir> --level N+1. The app is snapshotted to level-<N> before each upgrade.

Tip: run-loop.sh --backend mongodb --variant sequential-upgrade --level 12 automates the generate→grade→fix→upgrade cycle, but grading still happens in an interactive Claude session (Chrome MCP). For full manual control, drive 4a–4d yourself.

5. Grading (manual)

Grade only from observed browser behavior, never from source. Rubric:

Score	Meaning
3	Fully working as specified
2	Mostly working; minor bugs / missing edge cases
1	Partial; major issues
0	Missing or broken

Hard rules: JS console errors during a feature cap it at 2/3; real-time features that only work after a refresh cap at 1/3; untestable → 0; when in doubt, score lower.

Two identities (needed for typing / read receipts / unread / presence)

The app keys identity off localStorage, so two tabs in one profile = the same user. To get a real second user, either:

Incognito window as the second user (separate storage), or
A scripted second socket — register a second user via the API and open a separate socket.io connection (the method used in validation). Drive user 1 through the real UI; trigger user 2's typing / markRead / messages via the socket; grade user 1's UI reaction.

Open the app at the backend's port (mongodb: http://localhost:6373).

Record results

Write GRADING_RESULTS.md in the app dir (format in GRADING.md), one block per feature, plus a summary table ending in | **TOTAL** | **X/Y** | | — generate-report.mjs parses that line into the report's feature score.

File bugs

Copy templates/BUG_REPORT.template.md → <app-dir>/BUG_REPORT.md. One ## Bug N per issue, behavioral description, expected vs actual. That's the fix agent's input.

6. Reset DB between grading passes

To grade from a clean slate (no leftover users/rooms/messages):

bash

./reset-app.sh <app-dir>

Mongo: drops the database (Mongoose recreates collections on next write). Postgres: drops tables + drizzle-kit push. SpacetimeDB: publishes a fresh module.

7. Generate the comparison report

After a run (one or both backends in the same dated dir):

bash

node generate-report.mjs sequential-upgrade/sequential-upgrade-YYYYMMDD

Writes BENCHMARK_REPORT.md (cost, calls, tokens, duration, LOC; feature score if parseable). Aggregates per-backend cost from each session's cost-summary.json.

The investor-facing METRICS_DATA.json + public viewer have no in-tree generator — that aggregation is a separate manual/out-of-tree step. Treat BENCHMARK_REPORT.md as the local summary.

8. Cost tracking

Cost is captured automatically via OpenTelemetry — do not estimate tokens. Per session: telemetry/<run-id>/COST_REPORT.md (human) + cost-summary.json (structured). The dollar figure comes straight from Claude Code's cost_usd, so it's apples-to-apples across backends as long as the model is pinned the same.

Each run prepends a unique run-id to the prompt to bust the server-side prompt cache, so every run is a cold, fair measurement.

9. Teardown / start fresh

Smoke-test or aborted-run cleanup (all of this is regenerable / gitignored):

bash

cd tools/llm-sequential-upgrade
npx kill-port 6001 6373                                          # stop dev servers
docker exec llm-sequential-upgrade-mongodb-1 mongosh chat-app --quiet --eval "db.dropDatabase()"
rm -rf sequential-upgrade/sequential-upgrade-YYYYMMDD            # the run dir
: > telemetry/logs.jsonl && : > telemetry/metrics.jsonl         # shared telemetry

Leave the Mongo container + OTel collector up — the next run needs them. cleanup.sh <app-dir> (or --all) strips node_modules/dist/.git from an app dir without deleting the run.

10. Troubleshooting

Symptom	Check
`run.sh` exits at pre-flight	Is the DB container up? `docker compose … up -d`; for spacetime, `spacetime start`.
Port already in use	`npx kill-port <port>` (6001/6373 for mongodb).
Fix mode targets wrong port	Fixed — `run.sh` recomputes the Vite port after backend detection.
Mongo app misdetected as Postgres	The `.benchmark-backend` marker disambiguates; confirm it exists in the app dir.
OTel not capturing cost	`docker compose … logs otel-collector`; confirm `telemetry/logs.jsonl` is growing.
Report finds no telemetry	Point `generate-report.mjs` at the dated run dir, not a backend subdir.
Scripts fail under WSL/CI	Should be LF now (`.gitattributes`); re-checkout if you see `\r` errors.
Session ran out of context	Lower the level, or resume: `./run.sh --upgrade <dir> --level N --resume-session`.

Quick reference — a full MongoDB pass

bash

cd tools/llm-sequential-upgrade
docker compose -f docker-compose.otel.yaml up -d
export ANTHROPIC_MODEL=claude-sonnet-4-6

./run.sh --level 1 --backend mongodb          # → APP=<printed app dir>
#   grade at http://localhost:6373 → write GRADING_RESULTS.md (+ BUG_REPORT.md)
./run.sh --fix "$APP"                          # repeat with grading until clean
./reset-app.sh "$APP"                          # clean DB before re-grading

for L in 2 3 4 5 6 7 8 9 10 11 12; do
  ./run.sh --upgrade "$APP" --level $L
  #   grade → fix loop at each level
done

node generate-report.mjs sequential-upgrade/sequential-upgrade-$(date +%Y%m%d)