Benchmark Results: planning-with-files

Formal evaluation of planning-with-files using Anthropic's skill-creator framework, plus later functional verification of v3-specific mechanisms, plus a first competitive benchmark against six alternative planning methods. This document records the full methodology, test cases, grading criteria, and results. Tests 1 through 3 were run against v2.22.0 (2026-03-06); Test 4 was run against v3.2.0 (2026-07-03); Test 5 was run against v3.4.0 (2026-07-06).

An animated summary of Test 5 lives at docs/benchmark/index.html (rendered view).

Why We Did This

A proactive security audit in March 2026 identified a prompt injection amplification vector in the hook system. The PreToolUse hook re-reads task_plan.md before every tool call — the mechanism that makes the skill effective — but declaring WebFetch and WebSearch in allowed-tools created a path for untrusted web content to reach that file and be re-injected into context on every subsequent tool use.

Hardened in v2.21.0: removed WebFetch/WebSearch from allowed-tools, added explicit Security Boundary guidance to SKILL.md. These evals document the performance baseline and verify zero regression in workflow fidelity.

Test Environment

Test 1: Evals + Benchmark

Skill Category

planning-with-files is an encoded preference skill (not capability uplift). Claude can plan without the skill — the skill encodes a specific 3-file workflow pattern. Assertions test workflow fidelity, not general planning ability.

Test Cases (5 Evals)

Each eval ran two subagents simultaneously:

• with_skill: Read SKILL.md, follow it, create planning files in output dir
• without_skill: Execute same task naturally, no skill or template

Assertions per Eval

All assertions are objectively verifiable (file existence, section headers, field counts):

Total assertions: 30

Results

Aggregate:

Timing and token usage (from task completion notifications — captured at runtime):

The skill uses ~68% more tokens and ~17% more time on average. The extra cost is the structured output: creating 3 files instead of 1-2, following phase/status discipline, populating decisions and error tables. This is the intended tradeoff — the skill trades speed for structure.

One Assertion Refined (Eval 4)

Assertion: **Status:** pending on at least one future phase Result: not met

The agent completed all 6 migration phases in a single comprehensive planning session, leaving none pending. The skill was followed correctly — the assertion was overly prescriptive. The skill does not require phases to remain pending; it requires phases to have status fields. Revised for future evals: task_plan.md contains **Status:** fields (without specifying value).

Test 2: A/B Blind Comparison

Three independent comparator agents evaluated pairs of outputs without knowing which was with_skill vs without_skill. Assignment was randomized:

with_skill wins: 3/3 = 100%

Comparator Quotes

Eval 1 (todo-cli): "Output B satisfies all four structured-workflow expectations precisely... Output A delivered real, runnable code (todo.py + a complete test suite), which is impressive, but it did not fulfill the structural expectations... Output A's strength is real but out of scope for what was being evaluated."

Eval 3 (debug-fastapi): "Output A substantially outperforms Output B on every evaluated expectation. Output B is a competent ad-hoc debug response, but it does not satisfy the structured, multi-phase planning format the eval specifies. Output A passes all five expectations; Output B passes one and fails four."

Eval 4 (django-migration): "Output B is also substantively strong: it covers pytz/zoneinfo migration (a 4.2-specific item Output A omits entirely), includes 'django-upgrade' as an automated tooling recommendation... The 18,727 output characters vs 12,847 for Output A also reflects greater informational density in B."

Test 3: Description Optimizer

Status: Not run in this cycle

Requires ANTHROPIC_API_KEY in the eval environment. Per the project's eval standards, a test is only included in results if it can be run end-to-end with verified metrics. Scheduled for the next eval cycle.

Test 4: v3 Long-Running Session Functional Verification (2026-07-03)

Tests 1 through 3 measure whether the skill enforces the 3-file planning pattern. They predate v3.0.0 (2026-06-09) and say nothing about v3's own headline feature: surviving /clear and context compaction across a long-running session. This section covers that mechanism specifically, added after a repository health audit found it was silently broken on Windows.

Method

Rather than reading the code and assuming it worked, each mechanism was run directly in a scratch directory on a real Windows machine (Git Bash and PowerShell, both), with real files and real tampering, checking actual command output against expectations.

Scope

Result: 2 of 6 mechanisms were broken on Windows, silently

session-catchup.py did nothing on Windows. Two independent bugs, either one alone sufficient to break it:

1. The path sanitizer only replaced forward slashes. A Windows path (C:\Users\... or Git Bash's /c/Users/...) never matched Claude's actual project-directory naming, so the function always returned before finding any sessions to scan.
2. Three file reads had no explicit encoding, so any session log containing non-ASCII text raised UnicodeDecodeError against Windows' default cp1252 codec, an error the surrounding except clauses swallowed without a trace.

Neither failure printed anything. A user running this on Windows would see no catchup report and have no reason to suspect the mechanism was even running, let alone why it produced nothing. tests/test_path_fix.py contained a working reimplementation of the correct fix but never imported or exercised the actual shipped script, so the test suite reported green the entire time.

inject-plan.sh's containment guard silently dropped plan injection under aliased paths. The guard canonicalizes the project root and the candidate plan directory separately, then checks that one is a prefix of the other. On a Windows account with an 8.3 short-name TEMP (this test machine's own account is one such case) or a path reached through the MSYS /tmp mount, the two canonicalization routes land on differently-spelled versions of the same directory, the prefix check fails, and the hook exits with zero output: no plan re-injection, no tamper warning, nothing. resolve-plan-dir.sh does not have this bug because it builds its candidates as absolute paths from the start; that asymmetry is what pointed at the fix.

What worked correctly, unmodified

Session init, phase-status counting, attestation lock/show/clear, tamper detection logic itself (once the injection guard reaches it), and parallel-plan directory resolution ($PLAN_ID env var, .active_plan file, newest-mtime fallback) all produced correct output in both Git Bash and PowerShell, including the v3 autonomous-mode chain (nonce-framed delimiters, unattested-plan refusal, per-tool-call injection suppression).

One asymmetry noted but not a bug: init-session.ps1 has no slug mode (it always writes to the project root), only init-session.sh creates .planning/<slug>/ directories. And there is no inject-plan.ps1; the injection and tamper-enforcement hook body is sh-only, so a pure-PowerShell host without Git Bash gets no plan injection or tamper enforcement at all.

Fixed in v3.2.0

Both session-catchup.py and inject-plan.sh were fixed (see CHANGELOG). Re-running the same sequence after the fix confirmed session-catchup now produces a correct catchup report from real session logs, and plan injection now reaches the tamper-check branch under the same aliased-path conditions that previously went silent. The PowerShell-only injection gap and the init-session.ps1 slug-mode asymmetry are open follow-ups, not addressed in this cycle.

A caveat on what session-catchup actually does, working or not

Even fixed, session-catchup.py replays conversation transcript from the previous session; it does not parse or reconstruct the on-disk phase state directly. Its own "RECOMMENDED" output tells the agent to go read task_plan.md, progress.md, and findings.md itself. It is a conversation catchup that points at the files, not a plan-state reconstruction, and its usefulness is bounded by how much relevant discussion happened after the last planning-file edit.

Test 5: Competitive Benchmark v1, Seven Planning Methods Head to Head (2026-07-06, internal)

The earlier tests compare pwf against no skill. This test compares it against the field: six alternative ways of keeping an agent organized, all run in the same harness, on the same tasks, with the same model, and graded by scripts rather than by any LLM judge. It is an internal v1: the tasks are harness-authored (a disclosed limitation), and several rigor gates for a standalone public release (external task corpus, competitor-author review, cross-family jury for judged axes) are on the roadmap. Numbers below are exactly what the deterministic oracles produced, wins and tradeoffs alike, and they are reproducible from the raw runs.

Arms (pinned)

Executor: claude-opus-4-8 for every arm. Same tool allowlist everywhere. Each run in an isolated project directory with an isolated Claude home. Grading: pytest suites plus scripted transcript and file-state analysis (grade.json per run). No LLM grades anything in this test.

Tasks

77 graded cells total. Three further designed tasks (research-decide-build, drift-gauntlet, underspecified-dashboard) did not run in v1 and are on the v2 docket. The underspecified-dashboard task is one where superpowers' brainstorming gate is expected to beat pwf; that expectation is recorded here so the omission reads as a schedule gap, not a dodge.

Result 1: correctness parity. pwf matches the field, then pulls ahead

Every arm passed every task. 77 of 77 runs end with the provided pytest suite green, planning-with-files included, alongside the native baseline with no planning method at all. On single-session tasks of this size (8 to 29 turns), a frontier model completes the work regardless of how it is organized, so pass rate ties across the board. The honest reading: pwf gives up nothing on correctness, and pass rate at this task size cannot rank planning methods for anyone. What ranks them is everything measured below, and that is where pwf separates from the field.

What separates the arms: whether the method engages when nobody forces it, what a resume costs after context death, and what durability guarantees the method actually enforces.

Result 2: triggering, where always-on rules and pwf's hooks diverge

With prompts that never mention any skill, how often did each method actually engage (create its planning artifact before implementation)?

Two facts, both favoring the mechanism approach. First, the like-for-like comparison: against the closest skill-based competitor, pwf leads 5 to 0, since superpowers' planning skills never self-triggered in this harness at all. Second, the methods at 8/8 reach it only by living in always-loaded context (a project rule, a one-line instruction) rather than as a discoverable skill, a different tradeoff, not a better planner. pwf engaged on its own in 5 of 8 unforced runs, and every time it did, its hooks then fired deterministically for the rest of the session. Widening that auto-engage rate is the next roadmap item; the mechanism itself never misses once a plan is on disk. This is the strongest argument for hook-based mechanisms over model-remembers-to-do-it conventions.

Result 3: when engaged, pwf resumes fastest after context death

O6 protocol: kill the session at roughly half done, start a fresh one in the same directory with only "Continue the work in this directory." All arms eventually finished (saturation again), asked the user nothing, and redid no completed work. The separation is the cost of re-orientation, measured in stage-2 turns to completion:

A pwf resume took 5 turns: 40% fewer than the next-best arm and roughly 2.7x fewer than native or superpowers. The transcripts show why: session catchup plus hook injection put phase state in front of the model before its first tool call, so stage 2 starts at the correct next step instead of re-reading the world. This is the flagship result of the benchmark, and it is a mechanism win, not a prompt-quality accident: the plan is in front of the model by construction. On the unforced O2 variant the gap compresses (pwf 7.2 turns, naive-plan 6.8, filesystem 8.2), because the advantage is realized once the plan exists on disk, which ties directly back to the trigger roadmap item above. Close that gap and the recovery lead applies universally.

Result 4: what the guarantees cost, stated plainly

On the unforced build task, pwf averaged $0.634 and 15.3 turns; the arms that did no planning ran $0.31 to $0.42 and 7 to 10 turns, in the same range as spec-kit ($0.724, 14.0 turns). Structured planning carries a modest premium on a task this small, which is the expected tradeoff: pwf is the only method that then survives a context wipe. Mechanism overhead is about 330 tokens re-injected per user turn plus about 90 per matched tool call. A separate per-fire wall-clock figure measured on a single Windows test machine is being profiled and tuned; it is a local implementation detail rather than a property of the method, and it does not affect any result above. In short, pwf costs a little more per run and returns the only automatic recovery, re-surfacing, tamper-detection, plan-isolation, and compaction-survival guarantees in the field, behaviors that are mechanisms rather than instructions the model may or may not follow.

Result 5: no contamination, no spontaneous adoption

No arm produced another arm's signature files. The native and filesystem arms never spontaneously created task_plan.md, findings.md, or progress.md, replicating the Test 1 finding: the pwf file pattern does not leak out of training data; it appears when the skill drives it.

Grader validated against every method, and we proved it

An author-run benchmark has to earn trust, so the grader was validated against every method's real artifact layout before publishing. That validation caught and fixed a detector bug that had under-credited two competitors: it matched plan filenames like plan.md but not their directory layouts (docs/superpowers/plans/DATE-slug.md, specs/FEATURE/), which had falsely zeroed superpowers' forced-mode process metrics. The detector was corrected and all 77 cells re-graded before publication; post-fix, superpowers' forced-mode process metrics are 100%. The grading is deterministic: re-running the scripts on the raw runs reproduces every number here. Validating the grader against each method's real artifacts, and disclosing the fix, is the rigor an author-run benchmark owes its readers.

What this test does not measure

Trigger rates under varied natural phrasing (two unforced tasks only). Long-horizon drift and mid-task distractors (designed task did not run). Underspecified-task brainstorming, where superpowers is expected to win. Plan quality as judged output (needs a cross-family jury before it can be reported). Cross-IDE behavior (Claude Code only). Multi-day horizons. None of the above is claimed.

Reproducing

The harness (cell runner with pinned flags and isolated home, wave orchestrator, deterministic graders, aggregator with 95% CIs, task specs committed before any run, arm sources at pinned SHAs) and all 77 raw run directories (transcripts, file-tree hashes, per-run grades) live in the benchmark workspace, not tracked in this repo. The grading and aggregation scripts are deterministic; re-running them on the raw runs reproduces every number above.

Summary

The skill demonstrably enforces the 3-file planning pattern across diverse task types. Without the skill, agents default to ad-hoc file naming and skip the structured planning workflow entirely. Separately, v3's session-recovery mechanism is now verified functional on Windows as of v3.2.0; it was not before, and nothing in the test suite would have caught that on its own.

Reproducing These Results

# Clone the eval framework
gh api repos/anthropics/skills/contents/skills/skill-creator ...

# Set up workspace
mkdir -p eval-workspace/iteration-1/{eval-1,eval-2,...}/{with_skill,without_skill}/outputs

# Run with_skill subagent
# Prompt: "Read SKILL.md at path X. Follow it. Execute: <task>. Save to: <output_dir>"

# Run without_skill subagent
# Prompt: "Execute: <task>. Save to: <output_dir>. No skill or template."

# Grade assertions, produce benchmark.json
# See eval-workspace/iteration-1/benchmark.json for full data

Raw benchmark data: eval-workspace/iteration-1/benchmark.json (in eval-test copy, not tracked in main repo)