Intermittent / Flaky Failure Triage

Read this when a failure does not reproduce on every run — "fails sometimes", "a different test each run", "passes in isolation", "only fails in CI". Do not enter the hypothesis loop blind: intermittence itself is evidence. This triage usually collapses the search space in one round, before Phase 2 hypothesis formation.

---

Step 1 — Capture the failure signature (three cheap reruns)

Run these before forming any hypothesis. Journal each result.

Rerun	Command shape	Question it answers
Same scope, same command	exactly what just failed	Does the SAME test fail again, or a different one?
Failing test in isolation	single file / test filter	Does it pass alone?
Full scope on a quiet machine	stop concurrent builds, suites, background agents first	Does the whole suite go green when nothing else runs?
(When order is suspect) shuffled	-shuffle=on / pytest-randomly / runner's random seed	Does a specific order reproduce it? Record the seed.

Step 2 — Read the signature

Signature	Dominant hypothesis	Next move
Same test fails intermittently, everywhere	Real race in the code under test, or an async test bug (fixed sleep, unawaited promise, poll-for-time)	Standard phase loop from 02-investigate.md; reproduce deterministically by subscribing to the completion event or injecting a clock
A different test/file each run; each passes in isolation; quiet machine is green	Environment contention between concurrent runs — not the code	Step 3 checklist
Always the same test, but only in the full suite; isolation green	Test-order dependence / fixture leak (unreset module state, leaked env var, shared singleton)	Bisect with the shuffle seed; find the test that leaks, fix its teardown
Fails only on CI, never locally	Resource ceilings (slower disk/CPU → timeouts), different parallelism defaults, container clock	Reproduce locally under constraint (taskset, low --maxWorkers); raise the signal not the sleep
Green on plain rerun of the same commit	Still a flake — classify it with this table before ignoring it; an unclassified flake is a hidden bug report	Step 1 again with journaling

Step 3 — Concurrent-run contention checklist

The classic modern cause: two checkouts or worktrees of the same repo — or one checkout plus a background agent — running suites at the same time while sharing a global mutable resource. Multi-agent workstations make this the default failure mode, not an exotic one. Check each:

• Shared tmp roots — a sandbox/cache dir derived from a fixed tmpdir path ($TMPDIR/<project>-fixed-name) instead of a per-run mktemp. Both runs read/write the same tree; the loser sees half-deleted state.
• Fixed ports — hardcoded listen ports in test servers. Two suites race to bind; the loser gets 404/ECONNREFUSED mid-run, in whichever file happened to be running.
• Global env / config mutation — tests writing process-external state (dotfiles, shared config, global env) the other run reads.
• Shared containers / databases — same-named containers, same schema, same volume.
• Caches and locks — package-manager caches, lockfiles, .git index contention.

Evidence to capture while it is live: lsof -i :<port> during the failure window, a listing of the shared tmp path, a process list showing the concurrent runners.

Step 4 — Fix policy

• Environment contention → fix the test infrastructure, never the individual test: namespace every suite-global resource per run — mktemp for roots, port 0/ephemeral allocation, unique container names. The bar: two checkouts of this repo running the suite concurrently must not interfere. (The programming skill's test-isolation rules own this bar.)
• Order dependence → find the leaking fixture via the recorded shuffle seed; reset at teardown; re-run shuffled until the seed class is clean.
• Real race → normal phase loop (02-investigate.md onward); the fix ships with a failing-first test that reproduces deterministically.
• Forbidden regardless of cause: retry wrappers around the test, enlarged sleeps, .retry(N), quarantining, or deleting the test. Each one buries a bug report.

A flake you classified but did not fix is a finding to report, not to hide: journal the signature, the evidence, and the classification, and say it in your final report.