showcase/QA-COVERAGE.md
This matrix tracks what testing exists for each demo and the Sales Dashboard starter hero across manual QA checklists, automated unit tests, Playwright E2E tests, aimock-backed deterministic tests, and CI automation.
Legend:
| Demo | Manual QA | Vitest Unit | Playwright E2E (smoke) | Playwright E2E (interaction) | Per-Package E2E | Aimock Fixtures | CI Auto |
|---|---|---|---|---|---|---|---|
| Agentic Chat | PASS 17 packages | FAIL | PASS load + suggestions | WARN suggestion click only | PASS weather card, background change, multi-turn | WARN background, weather matches | WARN validate only (no Playwright in CI by default) |
| Human in the Loop | PASS 17 packages | FAIL | PASS load + suggestions | WARN suggestion click only | PASS step selector, approve/reject | WARN plan/steps/mars matches (text only, no interrupt) | WARN validate only |
| Tool Rendering | PASS 17 packages | FAIL | PASS load + suggestions | WARN suggestion click only | PASS WeatherCard with stats grid | WARN weather match (tool call) | WARN validate only |
| Gen UI (Tool-Based) | PASS 17 packages | FAIL | FAIL | FAIL | PASS sidebar, haiku card, pie/bar chart | FAIL no haiku-specific fixture | WARN validate only |
| Gen UI (Agent) | PASS 17 packages | FAIL | FAIL | FAIL | PASS task progress tracker, progress bar | FAIL no gen-ui-agent fixture | WARN validate only |
| Shared State (Read) | PASS 17 packages | FAIL | FAIL | FAIL | PASS recipe card, sidebar, pipeline | FAIL no shared-state fixture | WARN validate only |
| Shared State (Write) | PASS 17 packages (stub) | FAIL | FAIL | FAIL | PASS pipeline, deal CRUD, agent state writes | FAIL no shared-state fixture | WARN validate only |
| Shared State (Streaming) | PASS 17 packages (stub) | FAIL | FAIL | FAIL | PASS document editor, confirm/reject changes | FAIL no streaming fixture | WARN validate only |
| Sub-Agents | PASS 17 packages (stub) | FAIL | FAIL | FAIL | PASS travel planner, agent indicators, sections | FAIL no subagent fixture | WARN validate only |
| Feature | Manual QA | Vitest Unit | Playwright E2E (smoke) | Playwright E2E (interaction) | Aimock Fixtures | CI Auto |
|---|---|---|---|---|---|---|
| Sales Dashboard (page load) | FAIL | PASS extract-starter tests (on-demand extraction) | PASS header, 4 renderer pills | PASS pill switching, content verification | WARN sales/todo/deal matches | WARN validate + aimock-e2e (manual trigger) |
| Renderer Selector | FAIL | FAIL | PASS 4 pills visible, default selection | PASS mutual exclusion, content changes per mode | FAIL | WARN validate only |
| Tool-Based mode | FAIL | FAIL | PASS pipeline heading, KPI cards | PASS Add a deal, multiple deals, empty state | WARN sales/todo matches | WARN validate only |
| A2UI Catalog mode | FAIL | FAIL | PASS same pipeline content | FAIL | FAIL | WARN validate only |
| json-render mode | FAIL | FAIL | PASS fallback note + pipeline | FAIL | FAIL | WARN validate only |
| HashBrown mode | FAIL | FAIL | PASS pipeline content | FAIL | FAIL | WARN validate only |
showcase/integrations/*/qa/*.md)showcase/scripts/__tests__/*.test.ts)generate-registry.test.ts -- registry generationvalidate-constraints.test.ts -- constraint validationbundle-demo-content.test.ts -- demo content bundlingcreate-integration.test.ts -- integration creationshowcase/scripts/__tests__/e2e/)starter-e2e.spec.ts -- Sales Dashboard starter (15 tests: pills, modes, content switching, add deals)demo-e2e.spec.ts -- agentic-chat, hitl-in-chat, tool-rendering only (9 tests: load, suggestions, click)screenshots.spec.ts -- screenshot captureshowcase/integrations/*/tests/e2e/)langgraph-python additionally ships a 10th spec (renderer-selector.spec.ts) covering the Sales Dashboard renderer-selector flow; no other package has this spec.test_e2e-showcase-on-demand.yml trigger.showcase/aimock/)feature-parity.json -- 35 fixture entries covering: weather, charts, meetings, sales, flights, theme, background, plans, greetingssmoke.json -- minimal smoke test fixtures.github/workflows/showcase_*.yml)showcase_validate.yml -- runs npx vitest run on PR (unit tests only)test_e2e-showcase-on-demand.yml -- runs aimock-backed Playwright E2E, manual trigger only (/test-aimock comment or workflow_dispatch)showcase_drift-detection.yml -- template drift detectionshowcase_template-drift.yml -- template synchronizationshowcase_deploy.yml -- deployment pipelinedemo-e2e.spec.ts for the 6 demos currently missing (gen-ui-tool-based, gen-ui-agent, shared-state-read, shared-state-write, shared-state-streaming, subagents)