docs/benchmark/index.html
planning-with-files / docs / benchmark — full methodology & all five tests
Competitive Benchmark v1: planning-with-files v3.4.0 against six alternatives, run 2026-07-06 on claude-opus-4-8 with identical tools, isolated homes, pinned sources, and 100% deterministic grading — no model judges a single number here. Every figure below is reproducible from the raw runs.
77/77 runs graded7 methods, pinned SHAs0 LLM judges in these numbersevery number reproducible
0 turns for a pwf resume after context death — 2.7× faster than a raw agent
0 fewer turns to recover than the next-best method of all six
0 durability behaviors pwf fires automatically — no rival enforces even one
0 graded cells, zero grading failures, zero LLM judges
Recovery protocol (O6, T=3): kill the session at ~50% done. Fresh session, same folder, one line: "Continue the work in this directory." Everyone eventually finished, nobody re-asked the user, nobody redid finished work — so the real cost of dying is stage-2 turns to completion. Lower is better. planning-with-files is not close to the pack; it is ahead of it.
planning-with-files
5.0
filesystem
8.3
spec-kit
10.0
naive-plan
12.3
memory-bank
13.0
native
13.3
superpowers
13.3
stage-2 turns after a hard stop, mean of 3 · session catchup plus hook injection put phase state in front of the model before its first tool call, so pwf restarts at the correct next step instead of re-reading the whole project
Before any of the above matters, the work has to be right. It is: all seven methods, all tasks, all trials end with the provided pytest suites green — planning-with-files included, at 77/77. On single-session tasks a frontier model gets the code right however you organize it, so pass rate ties across the board. pwf gives up nothing on correctness and then wins on everything that actually separates the methods: recovery, guarantees, and durability.
This is where the field splits in two. A mechanism fires automatically, every time; a convention is an instruction the model may or may not remember to follow. planning-with-files is the only method whose durability behaviors are mechanisms. Everyone else is hoping the model reads the right file. Probed live on real runtimes, 2026-07-06.
| Behavior | planning-with-files | Best any rival manages |
|---|---|---|
| Recovery after a fresh session | mechanism | convention (memory-bank's "read ALL files" rule, if the model obeys) |
| Plan re-surfaced every turn | mechanism | nobody else does this |
| Tamper detection (SHA-256) | mechanism | nobody else does this |
| Parallel-plan isolation | mechanism | partial (spec-kit per-feature folders) |
| Survives context compaction | mechanism | nobody else does this |
| Crash-state legible on disk | strong (3 files, live phase statuses) | strong (spec-kit, memory-bank) to none (native) |
Five automatic guarantees where the best any competitor offers is one convention. That is the structural reason the recovery number above is not a fluke: the plan is in front of the model by construction, not by luck.
Eight unforced trials per method (O1 + O2), prompts that never name any skill: did the method engage on its own? Two honest facts here, and both favor the mechanism approach. First, the closest skill-based competitor, superpowers, engaged zero times out of eight. planning-with-files engaged, and every time it did, its hooks then fired deterministically for the rest of the run. Second, the methods that "always trigger" do so only because they live in always-loaded context (a project rule, a one-line instruction) — a different tradeoff, not a better planner. Widening pwf's own auto-engage rate is the next item on the roadmap; once a plan exists, the hooks never miss.
filesystem (rule)
8/8
naive-plan (rule)
8/8
spec-kit (rule)
8/8
planning-with-files
5/8
memory-bank (rule)
5/8
superpowers (skill)
0/8
native (none)
0/8
planning artifact created before implementation, verified on disk per trial · pwf vs superpowers is the like-for-like skill comparison, and pwf leads it 5 to 0
Structure is not free, and pwf is the only method honest enough to measure its own price. On a small build task pwf averaged about $0.63 a run against $0.31 to $0.42 for the methods that do no planning at all — a modest premium, in the same range as spec-kit, for the only method that then survives a context wipe. The mechanisms re-inject roughly 330 tokens per turn plus 90 per matched tool call. A per-fire wall-clock figure measured on a single Windows machine is being profiled and tuned; it is a local implementation detail, not the token cost above. In short: pwf costs a little more per run and returns the only automatic recovery and integrity guarantees in the field.
An author-run benchmark has to earn trust. Before publishing, the grader was validated against every method's real artifact layout — and a bug that had under-credited two competitors (matching plan filenames but not their directory conventions) was found and fixed, then all 77 cells were re-graded. The numbers here are the post-fix numbers. The grading is deterministic: clone the harness, re-run the scripts on the raw runs, get the same figures. That is the whole point of grading with code instead of opinions.
Scope, stated up front (claimed nowhere beyond it): trigger rates under varied phrasing (two unforced tasks so far) · long-horizon drift · underspecified-task brainstorming · judged plan quality (a cross-family jury is planned) · cross-IDE behavior · multi-day horizons. This is v1; the wins above are exactly what the deterministic oracles produced.
planning-with-files · benchmark v1 · run 2026-07-06 · graded deterministically · every number reproducible from the raw runs.