Rust E2E CI Reporting

Rust owns behavioral case declarations, observations, result classification, and report validation. OS runners build the environment, execute Rust targets, and upload the validated artifacts.

Canonical Invocation

The contributor-facing invocation runs the complete matrix on every OS and takes no suite selector:

Windows: scripts/ci/windows/run-rust-e2e.ps1 -RequireGui
Linux:   scripts/ci/linux/run-rust-e2e.sh
macOS:   packages/cua-driver/tests/runners/macos-lume/run-all.sh

Automated workflows may set a private lane value to fan that matrix into shared, native, and capture jobs so one failure does not hide another lane. The maintainer-run macOS Lume gate runs the complete matrix by default. These are execution partitions, not public suite choices or separate behavioral sources of truth.

Execution Order

Each runner follows the same sequence:

1. Resolve one immutable source SHA and check out that SHA in every lane.
2. Build the Rust driver and required repo-local fixtures.
3. Run e2e_environment_preflight_test once; it verifies git rev-parse HEAD or the VM source marker against the resolved SHA. On macOS it also verifies that the installed, TCC-authorized daemon embeds that exact SHA.
4. Abort before behavioral cells if the desktop, permissions, fixture, capture, or video lifecycle is unavailable.
5. Run the selected Rust integration targets.
6. Validate declarations, results, and evidence with cua-e2e-report.
7. Upload or retrieve the report, logs, and recordings.

The preflight prevents a single TCC or desktop-session problem from appearing as the same failure on every behavioral cell.

Artifact Files

Each OS lane writes:

artifacts/cua-driver/<platform>/
|-- environment.jsonl
|-- cases.jsonl
|-- results.jsonl
|-- summary.md
|-- environment-preflight.log
|-- <rust-target>.log
`-- recordings/<cell-label>-pid<process>-<sequence>/
    |-- recording.mp4
    |-- trajectory.json
    |-- session.json
    |-- cursor.jsonl
    `-- turn-*/
        |-- action.json
        |-- evidence.json
        |-- before_state.json
        |-- before.png
        |-- after_state.json
        |-- after.png
        |-- app_state.json
        |-- screenshot.png
        `-- click.png (dispatched click-family turns)

cases.jsonl is the executed catalog. results.jsonl contains one result for every declared cell. In canonical mode, the reporter rejects duplicates, missing results, undeclared results, contradictory statuses, forbidden skips, an under-sized unfiltered shared catalog, missing required videos, and missing or invalid expected turn evidence. Canonical runners set CUA_E2E_FORBID_SKIPS=1 and CUA_E2E_EXPECTED_MIN_CELLS (80 on Windows/Linux, 120 on macOS). Explicit diagnostic filters omit only the minimum-count policy; a filter matching zero cells still fails in Rust. An evidence.json classification normally makes capture failure auditable without making it pass. The one narrow exception is a successful Windows bring_to_front restoration: the before image may be target_minimized, and the after image may be capture_failed, only when the action reports success, after-state accessibility data was captured, and the required trajectory and finalized MP4 remain present. Any other missing image still fails closed.

A click-family call refused before its target can be resolved is the other explicit non-capture case: action.json records result_error: true without a click_point, and evidence.json records the click phase as not_applicable/action_refused_before_target_resolution. No input was aimed or dispatched, so that turn must not invent a click.png; successful and target-resolved click-family turns still require it.

The testkit prepares each cell's artifact directory before fixture setup but does not start recording.mp4 yet. On Windows GitHub-hosted runners it obtains the test process's inherited top-level console with GetConsoleWindow, verifies the HostedComputeAgent/runner title and console class, then minimizes it with ShowWindow(SW_MINIMIZE). Fixture readiness and the required foreground or sentinel-occluded posture are established after that cleanup. The cell then calls start_behavior_recording, re-establishes the sentinel posture after the capture backend attaches, and clears setup-only focus events. It dispatches the action only after this observation boundary, then keeps recording through oracle collection.

trajectory.json records the behavioral video phase as pending, started, or finalized, its start/baseline/finalization timestamps, and the hosted-runner console cleanup status. Strict reporting requires finalized. A setup or posture failure before the boundary remains in the test result and logs, and the testkit writes recording-error.txt; it does not create a misleading behavioral clip.

Environment Record

The preflight emits exactly one record:

{
  "schema": "cua-e2e-environment/v2",
  "platform": "macos",
  "display_server": "quartz",
  "source_sha": "0123456789abcdef0123456789abcdef01234567",
  "status": "ready",
  "duration_ms": 912,
  "message": ""
}

An error record produces an environment section in summary.md and aborts the lane before case declarations are executed. The same source SHA appears in the successful typed summary.

Case And Result Contract

A case declaration uses cua-e2e-case/v2. A result flattens the same contract fields into cua-e2e-result/v2 and adds the observation:

{
  "schema": "cua-e2e-result/v2",
  "cell_id": "windows-tauri-left-click-ax-background",
  "platform": "windows",
  "display_server": "win32",
  "harness": "tauri",
  "toolkit": "platform-webview",
  "action": "left_click",
  "targeting": "ax",
  "delivery": "background",
  "scope": "window",
  "driver_route": "uia_invoke",
  "expected_behavior": { "kind": "deliver" },
  "oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],
  "test_status": "pass",
  "observed_behavior": "delivered",
  "passed_oracles": ["fixture_state", "focus", "z_order", "no_leaked_input", "cursor"],
  "duration_ms": 1482,
  "message": "",
  "evidence": {
    "video": "recordings/windows-tauri-left-click-ax-background-pid123-001/recording.mp4",
    "trajectory": "recordings/windows-tauri-left-click-ax-background-pid123-001/trajectory.json"
  }
}

targeting describes how the action identifies its target: ax, px, page, or not_applicable. It is separate from screenshot/tree capture.

Classification Rules

test_status answers whether the cell met its declared contract. observed_behavior records what the driver did.

The structured refusal enum currently recognizes background_unavailable, background_occluded, and background_uipi_blocked. Each refusal case declares the exact subset allowed by its controlled setup. String-prefix or message-substring matching is not accepted.

A refusal contract also requires focus, z-order, and leaked-input observations. An honest refusal does not satisfy a case whose contract requires delivery.

Summary Ownership

cua-e2e-report is the only behavioral Markdown renderer. Shell and PowerShell runners must not parse test ... ok output or create host=cargo and host=lane rows. Cargo logs remain available for unit-test annotations and lane diagnostics, outside the behavioral population.

The summary reports delivered passes, refused passes, failures, and skips separately. Shared and native targets emit the same typed records; a compile or runner failure that prevents declaration remains visible as a failed job and cannot be mistaken for a green behavioral row.

Evidence Links

Every canonical behavioral cell records a validated MP4 and trajectory. The recording directory also contains its session metadata and turn-level action and screenshot files; Cargo target logs remain lane-level diagnostics. GitHub cannot deep-link to a file inside a multi-cell artifact archive, so each summary row displays the exact MP4 path as a link to the owning lane archive while the trajectory rollup provides bulk-download links and video counts.