E2E scenarios: the contract

With coding agents writing most of the production code, e2e tests are the document humans actually read and review. A scenario must state an intent, a compose model, and a sequence of steps (command → expected observables) — and nothing else. Everything operational (project naming, cleanup, failure diagnostics) belongs to the framework, not to the test.

The DSL lives in scenario.go (execution, actions, requirements) and checks.go (the vocabulary of observables).

Writing a scenario

The project files live in testdata/<TestName>/ — standalone compose files, directly runnable with docker compose -f testdata/TestRestart/compose.yaml:

# testdata/TestRestart/compose.yaml
services:
  app:
    image: alpine
    init: true
    command: ash -c "if [[ -f /tmp/restart.lock ]] ; then sleep infinity; else touch /tmp/restart.lock; fi"

func TestRestart(t *testing.T) {
	NewScenario(t, "restart must bring an exited service back up, restarting the same container").
		Step("up starts the service, whose first run exits at once",
			ComposeCmd("up", "-d"),
			Eventually(ServiceState("app", "exited"), 10*time.Second)).
		Step("restart brings the service back up, reusing the container",
			ComposeCmd("restart"),
			Eventually(ServiceState("app", "running"), 10*time.Second),
			NotRecreated("app"))
}

Rules:

• New e2e tests use NewScenario. The legacy NewCLI style remains for existing tests, converted opportunistically; don't add to it.
• One intent = one invariant. The intent is a one-line statement of the behavior being locked, phrased as an obligation ("X must Y"). If you need two intents, write two scenarios.
• Step names are behavior sentences, not command echoes: "an unchanged create is a no-op", not "run create again". The transcript of step names should read as the specification.
• The project files live in testdata/<TestName>/. The scenario resolves that directory by convention (subtests map to nested directories, following t.Name()) and copies it to a temporary directory, so the committed files are never mutated. The directory holds a compose.yaml plus whatever the project needs (Dockerfile, env or config files) in their native format, directly runnable outside the test. Ownership is strictly one test per directory — no shared fixtures — and a check fails the suite on any testdata directory no test owns. Interpolate runtime values via Env.
• Regression tests link the issue in a comment above the test, with a sentence on the failure mode being locked.

Checks: observe real state

Checks are the shared vocabulary between scenarios; their discipline is what keeps the contract meaningful.

• Prefer state-based checks (ServiceState, NotRecreated, LabelSet, RunsOnPlatform, …): they observe containers, labels and image manifests — what the user actually gets — not what the CLI printed.
• OutputContains is a last resort, legitimate only when the CLI's reported decision is itself the observable (e.g. "Skipped" vs "Pulled").
• Never poll by hand: wrap a state check in Eventually(check, timeout). No time.Sleep in scenarios.
• A new check must be generic — no test-specific logic — and named after the observable it asserts, not after the test that needed it. It goes in checks.go, where the whole vocabulary is reviewed as one file. Before adding one, verify the observable isn't already expressible.
• A check should also fail loudly on a broken precondition (e.g. NotRecreated errors if the service had no container before the step) rather than pass vacuously.

When a scenario fails

The report opens with everything needed to diagnose without re-running:

• artifacts: <dir> — a stable per-project directory holding the untruncated material: compose.yaml, failure.txt, each step's full command and output (step-NN-*.txt), containers.txt, events.txt, full container logs (logs-*.txt) and the per-step state snapshots (snapshots.json). Read these before re-running anything.
• E2E_KEEP_FAILED=1 — rerun with this set to skip teardown of failed scenarios: containers, volumes and networks stay alive for docker inspect/exec. Clean up afterwards with docker compose --project-name <project> down -v --remove-orphans.
• The inline report shows the transcript (every step, exit code, duration), the failing step's output, project containers, engine events since the scenario started, and container log tails — truncated for readability; the artifacts have the full versions.

Run a single scenario with:

go test -tags e2e ./pkg/e2e/ -run TestRestart -v