Homologation Suite (`npm run homolog`)

Real-environment E2E validation of the OmniRoute deploy running on the homologation VPS (HOMOLOG_BASE_URL, e.g. http://192.168.0.15:20128). One command replaces the manual release STOP #2 checklist with an automated, evidence-producing run.

What it covers

Layer	What it checks	Implementation
L0 — health/parity	`/api/monitoring/health` responds `200` with `status: "healthy"` and the expected version	`scripts/homolog/lib/parity.mjs`
L1a — ephemeral key	Admin login → `POST /api/keys` creates a scoped API key for the run, revoked (`DELETE /api/keys/:id`) in a `finally` block regardless of outcome	`scripts/homolog/lib/adminClient.mjs`
L1b — API surface	`/v1/models` catalog, a real non-streaming chat completion (tier-critical model, `max_tokens: 5`), an invalid-key `401`, and public `/api/monitoring/health`	`tests/homolog/api/core.http` (httpYac)
L1c — SSE streaming	Real streaming chat completion; asserts `text/event-stream`, at least one content delta, and a `[DONE]` terminator	`scripts/homolog/lib/sseCheck.mjs`
L2 — real providers	One minimal-cost chat request per critical provider present in the live `/v1/models` catalog, generated on the fly via promptfoo	`scripts/homolog/gen-promptfoo.mjs` + `scripts/homolog/lib/providerTiers.mjs`
L4a — UI auth	Logs in once via the real login form and reuses the session (`storageState`) across the UI layer	`tests/homolog/ui/auth.setup.ts`
L4b — UI routes	Every static `page.tsx` under `src/app/(dashboard)/dashboard` (discovered from the filesystem, dynamic `[param]` routes skipped) loads without an HTTP error, a page error, or the Next.js error boundary	`tests/homolog/ui/routes.spec.ts`
L4c — UI critical flow	Creates an API key through the dashboard UI and revokes it again (leaves no residue on the VPS)	`tests/homolog/ui/api-key-flow.spec.ts`
L5 — unified report	Merges httpYac (via `junit-to-ctrf`), the promptfoo→CTRF adapter, and the Playwright CTRF reporter into one `homolog-ctrf.json`, plus a human-readable `homolog-report/summary.md`	`scripts/homolog/run.mjs`

Zero LLM involvement in the replay itself — this is a deterministic regression battery, not an eval. AI only enters in future maintenance work (see Roadmap below).

Prerequisites

Copy .env.homolog.example to .env.homolog (gitignored — never commit it) and fill in:
- HOMOLOG_BASE_URL — the target deploy, e.g. http://192.168.0.15:20128.
- HOMOLOG_ADMIN_PASSWORD — the dashboard management password for that deploy.
- HOMOLOG_CRITICAL_PROVIDERS — comma-separated provider prefixes that get a real smoke chat request (e.g. openai,anthropic,gemini,codex,grok,glm,deepseek,openrouter).
- HOMOLOG_API_KEY — leave empty in normal runs; the suite creates and revokes its own ephemeral key. Only set this to debug a single layer in isolation.
npm install in the repo (the suite's dependencies — httpyac, promptfoo, playwright-ctrf-json-reporter, junit-to-ctrf, ctrf — are regular devDependencies).
npx playwright install if the browser binaries are not already present.

How to run

bash

npm run homolog

To validate against a deploy whose version does not match the local package.json (e.g. a homologation box still on a previous patch release), override the expected version explicitly:

bash

HOMOLOG_EXPECT_VERSION=3.8.47 npm run homolog

The run exits non-zero if any layer fails, and always attempts to revoke the ephemeral API key it created, even on failure (finally block in scripts/homolog/run.mjs).

Reading the report

All output lands in homolog-report/ (gitignored):

summary.md — the same table printed to stdout, one row per layer (✅/❌ + detail).
homolog-ctrf.json — the unified CTRF report (merge of API/SSE, provider-smoke, and UI results) — this is the artifact to attach to a release STOP #2 checklist.
httpyac-junit.xml, api-ctrf.json, providers-ctrf.json, ui-ctrf.json — the per-layer raw/intermediate reports.
promptfooconfig.yaml, provider-misses.json — the generated promptfoo config for the current run and any critical providers that were missing from the live catalog.

A failing L0 aborts immediately (no ephemeral key is created) since a version/health mismatch means every downstream layer would be validating the wrong deploy.

Re-baselining when the UI changes legitimately

L4b (route smoke) and L4c (API-key UI flow) are driven by real DOM locators, not snapshots, so most legitimate UI changes do not require any suite update. When a change does break a locator (e.g. a renamed button label or a moved settings page):

Re-confirm the locator against the current source (the specs already document which file/line each locator was confirmed against — follow the same pattern, don't guess).
Update the spec in tests/homolog/ui/.
Re-run npm run homolog (or just the affected Playwright spec) against the VPS to confirm the fix, then commit.

There is no visual/pixel baseline in this suite (F1) — see Roadmap for that.

Roadmap (F2 / F3)

Design and phased rollout live in the internal planning spec _tasks/superpowers/specs/2026-07-13-homolog-e2e-suite-design.md (not linked — internal _tasks/ artifact, not part of this repo's tracked docs). Summary:

F2 — full walkthrough recording → Playwright Test Agents (planner/generator) turn it into flow specs (create combo, test provider, edit settings, MCP tools) + visual regression baseline (Lost Pixel) with masks over dynamic data (metrics, timestamps, logs) + a healer maintenance routine per release.
F3 — resilience/contract/wiring coverage: toxiproxy + a fake OpenAI-compatible provider on the devbox, a homolog-resilience combo on the VPS pointed at it (injected timeout → assert fallback + circuit breaker open/close via /api/monitoring/health); gated Schemathesis contract testing against docs/openapi.yaml (low --max-examples, fixed seeds, non-LLM endpoints only); and wiring npm run homolog + its summary.md into the /generate-release STOP #2 phase.

Homologation Suite (npm run homolog)