Docker Release Browser E2E Plan

Context

Today release smoke testing for published Paperclip packages is manual and shell-driven:

HOST_PORT=3232 DATA_DIR=./data/release-smoke-canary PAPERCLIPAI_VERSION=canary ./scripts/docker-onboard-smoke.sh
HOST_PORT=3233 DATA_DIR=./data/release-smoke-stable PAPERCLIPAI_VERSION=latest ./scripts/docker-onboard-smoke.sh

That is useful because it exercises the same public install surface users hit:

• Docker
• npx paperclipai@canary
• npx paperclipai@latest
• authenticated bootstrap flow

But it still leaves the most important release questions to a human with a browser:

• can I sign in with the smoke credentials?
• do I land in onboarding?
• can I complete onboarding?
• does the initial CEO agent actually get created and run?

The repo already has two adjacent pieces:

• tests/e2e/onboarding.spec.ts covers the onboarding wizard against the local source tree
• scripts/docker-onboard-smoke.sh boots a published Docker install and auto-bootstraps authenticated mode, but only verifies the API/session layer

What is missing is one deterministic browser test that joins those two paths.

Goal

Add a release-grade Docker-backed browser E2E that validates the published canary and latest installs end to end:

1. boot the published package in Docker
2. sign in with known smoke credentials
3. verify the user is routed into onboarding
4. complete onboarding in the browser
5. verify the first CEO agent exists
6. verify the initial CEO run was triggered and reached a terminal or active state

Then wire that test into GitHub Actions so release validation is no longer manual-only.

Recommendation In One Sentence

Turn the current Docker smoke script into a machine-friendly test harness, add a dedicated Playwright release-smoke spec that drives the authenticated browser flow against published Docker installs, and run it in GitHub Actions for both canary and latest.

What We Have Today

Existing local browser coverage

tests/e2e/onboarding.spec.ts already proves the onboarding wizard can:

• create a company
• create a CEO agent
• create an initial issue
• optionally observe task progress

That is a good base, but it does not validate the public npm package, Docker path, authenticated login flow, or release dist-tags.

Existing Docker smoke coverage

scripts/docker-onboard-smoke.sh already does useful setup work:

• builds Dockerfile.onboard-smoke
• runs paperclipai@${PAPERCLIPAI_VERSION} inside Docker
• waits for health
• signs up or signs in a smoke admin user
• generates and accepts the bootstrap CEO invite in authenticated mode
• verifies a board session and /api/companies

That means the hard bootstrap problem is mostly solved already. The main gap is that the script is human-oriented and never hands control to a browser test.

Existing CI shape

The repo already has:

• .github/workflows/e2e.yml for manual Playwright runs against local source
• .github/workflows/release.yml for canary publish on master and manual stable promotion

So the right move is to extend the current test/release system, not create a parallel one.

Product Decision

1. The release smoke should stay deterministic and token-free

The first version should not require OpenAI, Anthropic, or external agent credentials.

Use the onboarding flow with a deterministic adapter that can run on a stock GitHub runner and inside the published Docker install. The existing process adapter with a trivial command is the right base path for this release gate.

That keeps this test focused on:

• release packaging
• auth/bootstrap
• UI routing
• onboarding contract
• agent creation
• heartbeat invocation plumbing

Later we can add a second credentialed smoke lane for real model-backed agents.

2. Smoke credentials become an explicit test contract

The current defaults in scripts/docker-onboard-smoke.sh should be treated as stable test fixtures:

• email: [email protected]
• password: paperclip-smoke-password

The browser test should log in with those exact values unless overridden by env vars.

3. Published-package smoke and source-tree E2E stay separate

Keep two lanes:

• source-tree E2E for feature development
• published Docker release smoke for release confidence

They overlap on onboarding assertions, but they guard different failure classes.

Proposed Design

1. Add a CI-friendly Docker smoke harness

Refactor scripts/docker-onboard-smoke.sh so it can run in two modes:

• interactive mode
  • current behavior
  • streams logs and waits in foreground for manual inspection
• CI mode
  • starts the container
  • waits for health and authenticated bootstrap
  • prints machine-readable metadata
  • exits while leaving the container running for Playwright

Recommended shape:

• keep scripts/docker-onboard-smoke.sh as the public entry point
• add a SMOKE_DETACH=true or --detach mode
• emit a JSON blob or .env file containing:
  • SMOKE_BASE_URL
  • SMOKE_ADMIN_EMAIL
  • SMOKE_ADMIN_PASSWORD
  • SMOKE_CONTAINER_NAME
  • SMOKE_DATA_DIR

The workflow and Playwright tests can then consume the emitted metadata instead of scraping logs.

Why this matters

The current script always tails logs and then blocks on wait "$LOG_PID". That is convenient for manual smoke testing, but it is the wrong shape for CI orchestration.

2. Add a dedicated Playwright release-smoke spec

Create a second Playwright entry point specifically for published Docker installs, for example:

• tests/release-smoke/playwright.config.ts
• tests/release-smoke/docker-auth-onboarding.spec.ts

This suite should not use Playwright webServer, because the app server will already be running inside Docker.

Browser scenario

The first release-smoke scenario should validate:

1. open /
2. unauthenticated user is redirected to /auth
3. sign in using the smoke credentials
4. authenticated user lands on onboarding when no companies exist
5. onboarding wizard appears with the expected step labels
6. create a company
7. create the first agent using process
8. create the initial issue
9. finish onboarding and open the created issue
10. verify via API:
    • company exists
    • CEO agent exists
    • issue exists and is assigned to the CEO
11. verify the first heartbeat run was triggered:
    • either by checking issue status changed from initial state, or
    • by checking agent/runs API shows a run for the CEO, or
    • both

The test should tolerate the run completing quickly. For this reason, the assertion should accept:

• queued
• running
• succeeded

and similarly for issue progression if the issue status changes before the assertion runs.

Why a separate spec instead of reusing tests/e2e/onboarding.spec.ts

The local-source test and release-smoke test have different assumptions:

• different server lifecycle
• different auth path
• different deployment mode
• published npm package instead of local workspace code

Trying to force both through one spec will make both worse.

3. Add a release-smoke workflow in GitHub Actions

Add a workflow dedicated to this surface, ideally reusable:

• .github/workflows/release-smoke.yml

Recommended triggers:

• workflow_dispatch
• workflow_call

Recommended inputs:

• paperclip_version
  • canary or latest
• host_port
  • optional, default runner-safe port
• artifact_name
  • optional for clearer uploads

Job outline

1. checkout repo
2. install Node/pnpm
3. install Playwright browser dependencies
4. launch Docker smoke harness in detached mode with the chosen dist-tag
5. run the release-smoke Playwright suite against the returned base URL
6. always collect diagnostics:
   • Playwright report
   • screenshots
   • trace
   • docker logs
   • harness metadata file
7. stop and remove container

Why a reusable workflow

This lets us:

• run the smoke manually on demand
• call it from release.yml
• reuse the same job for both canary and latest

4. Integrate it into release automation incrementally

Phase A: Manual workflow only

First ship the workflow as manual-only so the harness and test can be stabilized without blocking releases.

Phase B: Run automatically after canary publish

After publish_canary succeeds in .github/workflows/release.yml, call the reusable release-smoke workflow with:

• paperclip_version=canary

This proves the just-published public canary really boots and onboards.

Phase C: Run automatically after stable publish

After publish_stable succeeds, call the same workflow with:

• paperclip_version=latest

This gives us post-publish confirmation that the stable dist-tag is healthy.

Important nuance

Testing latest from npm cannot happen before stable publish, because the package under test does not exist under latest yet. So the latest smoke is a post-publish verification, not a pre-publish gate.

If we later want a true pre-publish stable gate, that should be a separate source-ref or locally built package smoke job.

5. Make diagnostics first-class

This workflow is only valuable if failures are fast to debug.

Always capture:

• Playwright HTML report
• Playwright trace on failure
• final screenshot on failure
• full docker logs output
• emitted smoke metadata
• optional curl /api/health snapshot

Without that, the test will become a flaky black box and people will stop trusting it.

Implementation Plan

Phase 1: Harness refactor

Files:

• scripts/docker-onboard-smoke.sh
• optionally scripts/lib/docker-onboard-smoke.sh or similar helper
• doc/DOCKER.md
• doc/RELEASING.md

Tasks:

1. Add detached/CI mode to the Docker smoke script.
2. Make the script emit machine-readable connection metadata.
3. Keep the current interactive manual mode intact.
4. Add reliable cleanup commands for CI.

Acceptance:

• a script invocation can start the published Docker app, auto-bootstrap it, and return control to the caller with enough metadata for browser automation

Phase 2: Browser release-smoke suite

Files:

• tests/release-smoke/playwright.config.ts
• tests/release-smoke/docker-auth-onboarding.spec.ts
• root package.json

Tasks:

1. Add a dedicated Playwright config for external server testing.
2. Implement login + onboarding + CEO creation flow.
3. Assert a CEO run was created or completed.
4. Add a root script such as:
   • test:release-smoke

Acceptance:

• the suite passes locally against both:
  • PAPERCLIPAI_VERSION=canary
  • PAPERCLIPAI_VERSION=latest

Phase 3: GitHub Actions workflow

Files:

• .github/workflows/release-smoke.yml

Tasks:

1. Add manual and reusable workflow entry points.
2. Install Chromium and runner dependencies.
3. Start Docker smoke in detached mode.
4. Run the release-smoke Playwright suite.
5. Upload diagnostics artifacts.

Acceptance:

• a maintainer can run the workflow manually for either canary or latest

Phase 4: Release workflow integration

Files:

• .github/workflows/release.yml
• doc/RELEASING.md

Tasks:

1. Trigger release smoke automatically after canary publish.
2. Trigger release smoke automatically after stable publish.
3. Document expected behavior and failure handling.

Acceptance:

• canary releases automatically produce a published-package browser smoke result
• stable releases automatically produce a latest browser smoke result

Phase 5: Future extension for real model-backed agent validation

Not part of the first implementation, but this should be the next layer after the deterministic lane is stable.

Possible additions:

• a second Playwright project gated on repo secrets
• real claude_local or codex_local adapter validation in Docker-capable environments
• assertion that the CEO posts a real task/comment artifact
• stable release holdback until the credentialed lane passes

This should stay optional until the token-free lane is trustworthy.

Acceptance Criteria

The plan is complete when the implemented system can demonstrate all of the following:

1. A published paperclipai@canary Docker install can be smoke-tested by Playwright in CI.
2. A published paperclipai@latest Docker install can be smoke-tested by Playwright in CI.
3. The test logs into authenticated mode with the smoke credentials.
4. The test sees onboarding for a fresh instance.
5. The test completes onboarding in the browser.
6. The test verifies the initial CEO agent was created.
7. The test verifies at least one CEO heartbeat run was triggered.
8. Failures produce actionable artifacts rather than just a red job.

Risks And Decisions To Make

1. Fast process runs may finish before the UI visibly updates

That is expected. The assertions should prefer API polling for run existence/status rather than only visual indicators.

2. latest smoke is post-publish, not preventive

This is a real limitation of testing the published dist-tag itself. It is still valuable, but it should not be confused with a pre-publish gate.

3. We should not overcouple the test to cosmetic onboarding text

The important contract is flow success, created entities, and run creation. Use visible labels sparingly and prefer stable semantic selectors where possible.

4. Keep the smoke adapter path boring

For release safety, the first test should use the most boring runnable adapter possible. This is not the place to validate every adapter.

Recommended First Slice

If we want the fastest path to value, ship this in order:

1. add detached mode to scripts/docker-onboard-smoke.sh
2. add one Playwright spec for authenticated login + onboarding + CEO run verification
3. add manual release-smoke.yml
4. once stable, wire canary into release.yml
5. after that, wire stable latest smoke into release.yml

That gives release confidence quickly without turning the first version into a large CI redesign.