packages/cua-driver/docs/browser-tool-implementation-journal.md
This journal records implementation and validation evidence for the first-class browser tools. It intentionally omits machine identities, credentials, private URLs, profile paths, and raw DevTools endpoint identifiers.
get_browser_state read-only and make setup explicit.(pid, window_id) targets to browser tabs exactly or refuse all
mutation. Never promote a heuristic match into an action route.page tool compatible until the new browser routes have
sufficient release evidence.origin/main at 4e9b26ed3077089c1812a62909b82ac5c5ae2150
on local branch codex/browser-tool-v1.PageBackend
inversion: shared contracts and CDP behavior in cua-driver-core; process,
endpoint, native-window, and setup evidence in each platform crate.Browser.getWindowForTarget and Browser.getWindowBounds.page facade, and
keep browser_prepare to explicit endpoint attachment or a minimal
driver-owned debug launch._NET_WM_PID; require independent
agreement with endpoint ownership before treating it as exact.cua-driver-core and registered the same
schemas on every platform.get_browser_state strictly read-only. browser_prepare detects an
existing owned endpoint or returns an explicit setup/consent limitation; it
does not silently relaunch or modify a profile.cdp_debugging_port launch support on Windows and Linux, matching the
existing macOS launch surface.session argument authoritative while retaining the daemon's
hidden session mirror for MCP transport compatibility.-32601 for
Browser.getWindowForTarget. Rather than guessing, added a bounded
embedded_single_page route.native_cdp_window route based on the
browser window id and native/CDP geometry agreement.<platform>-electron-browser-tool-roundtrip to the established shared
web behavior matrix instead of creating a separate test framework.browser_prepare recognizes the owned endpoint, asserts the bounded
embedded_single_page route, snapshots the page, performs a trusted ref
click, verifies the external fixture journal, snapshots again, types through
the new editable ref, and proves the older ref returns browser_ref_stale
without changing state. It then navigates to about:blank, verifies the new
URL, and proves navigation invalidates the latest page ref.browser_type now requires a current ref and proves that the focused node is
editable before dispatch. It refuses instead of reporting success against
unproven page focus.cargo test -p cua-driver-core -p platform-macos: 170 core tests, 3 session
lifecycle tests, and 126 platform-macOS tests passed at the final revision.cargo test -p cua-driver --test protocol_schema_test --test schema_consistency_test --test protocol_session_test: 1 schema, 1
consistency, and 6 session protocol tests passed.cargo build -p cua-driver and the release build used by the documentation
generator passed.0cf2dd9e1372a3b2c11822975752ddd04fe0b931 narrows the macOS
fallback cardinality check to titled, non-empty browser surfaces. A second
titled surface still defeats the proof, and a second CDP page independently
defeats it in shared core. Both shapes have focused regression tests.install-local.sh using the stable
local signing identity. The daemon reported Accessibility, Screen Recording,
and live capture capability as granted after restart.a2e792a428308a80bb02a37ed4467ea073a8e8e5.a2e792a428308a80bb02a37ed4467ea073a8e8e5.a2e792a428308a80bb02a37ed4467ea073a8e8e5.cua-driver skill. The legacy page
tool remains available and is not presented as the preferred new mutation
route.| Plan phase | Browser-tool v1 result |
|---|---|
| Phase 0: binding feasibility | Accepted. Native Chromium uses exact CDP-window geometry; Electron uses the separately bounded single-page/single-native-window proof. Ambiguous targets refuse. |
| Phase 1: shared CDP foundation | Implemented in cua-driver-core, with loopback listener ownership and exact WebSocket-port attestation in all three platform adapters. |
| Phase 2: session-owned targets | Implemented. Target, tab, snapshot, and ref capabilities are opaque and session-scoped; session teardown removes the namespace. |
| Phase 3: read-only state | Implemented. Bind and snapshot modes share get_browser_state; neither performs setup or mutation. |
| Phase 4: preparation | Intentionally narrow. Existing owned endpoints are recognized; setup, restart, and profile changes remain explicit structured refusals rather than hidden side effects. |
| Phase 5: typed mutations | Implemented for navigation, trusted or explicitly synthetic click, and ref-bound editable typing, with mutation-time revalidation. |
Phase 6: legacy page migration | Deferred. The existing facade remains compatible and separate until adoption evidence supports migration. |
| Phase 7: release evidence and rollout | Partially complete. Canonical Electron evidence, generated reference, public Diataxis docs, and bundled skills are present. Standalone Chrome/Edge adversarial lanes and embedded-webview expansion remain deferred. |
The v1 milestone therefore satisfies the accepted local implementation scope,
including canonical macOS, Windows, X11, and native Wayland evidence. It does
not claim the later page migration or the broader real-browser release
matrix.
The v2 entries below supersede the completed items in this historical list.
page facade migration.browser_prepare launch.26c177d9 made browser-engine classification and route limitations
consistent across platforms. Unsupported engines remain discoverable but do
not advertise mutation.0295b881 added repository-owned standalone Chromium harness coverage.3287b44e added adversarial multi-tab and same-bounds multi-window cases.
The driver refuses ambiguity instead of selecting the first page or using a
title tie-break across different CDP window ids.c8fb38ef added refs through open shadow roots and same-process iframes.
Out-of-process iframes are attached only when the runtime exposes a proven
CDP session; event messages are demultiplexed from command responses.ca31f017 removed public consent/restart fields and marked
browser_prepare destructive and non-idempotent.browser-approve command and bound to pid plus profile request.isolated_new or isolated_named profile. It never copies, modifies,
restarts, or terminates the selected user profile.DevToolsActivePort and
socket ownership. Temporary profiles and their processes are reaped when the
owning session ends.launch_app are rejected on all
platforms so setup cannot bypass this boundary.1c4e3539 separated exact browser-route evidence from the broad shared
action matrix. Electron proves an exact CDP mutation roundtrip on each
supported OS. Tauri, WKWebView, WebKitGTK, and the common split-process
WebView2 shape prove side-effect-free browser_route_unavailable refusals
until their engine/native-host relationship can be bound exactly.5b484098 migrated the legacy page CDP path to the shared pooled,
event-aware transport without changing its first-page/URL-hint semantics,
output format, timeouts, or AppleScript/UIA/AT-SPI fallbacks.a6e4c6b8 places every Unix test-owned browser in its own process group and
reaps the complete tree after each row. This removed late Chrome children
that could contaminate the next exact-binding test.cua-driver-testkit passed 41 tests. The full six-row standalone Chromium
suite then passed in 66 seconds with no lingering harness browser process.browser_prepare deliberately does not attach DevTools to, copy, or restart
a person's existing profile. Authenticated-profile automation remains a
product/security decision rather than an implicit setup shortcut.ee8b75f3 split the standalone browser matrix into independent scenario
processes. A failed launch or contaminated desktop can no longer prevent the
remaining declared rows from producing evidence.083859f3, 5f3fca7a, and 3336b343 made temporary Chromium startup and
teardown deterministic. Every harness browser receives a clean profile and,
on Unix, an owned process group whose descendants are reaped when the row
ends.33c8d533 made the harness route explicit in every result instead of
inferring it from the expected outcome.c70f5986 creates additional tabs and same-bounds windows through the
harness-only CDP connection. This removes command-line handoff ambiguity
while leaving the public driver path unchanged. It also lets macOS E2E use a
dedicated daemon socket so another driver client cannot end the test
session.edc26080 made the loopback fixture server concurrent. Chromium idle
preconnections can no longer hold the only server worker and starve the real
page or journal request. A regression test opens an idle connection while a
second client completes a fixture request.418fb972 made the reporter deduplicate identical environment records from
independently executed rows while continuing to reject conflicting records.4fc4e00b fixed a critical Unix harness teardown bug. Passing -1 to the
external kill utility had been parsed as the broad kill(-1, SIGKILL)
syscall, which could terminate every process signalable by the test user.
Teardown now calls guarded libc::kill(-pgid, SIGKILL) directly and refuses
process ids 0, 1, or values outside the platform range.edc2608061360dadc258724045438922487753e2:
16 declared Chrome and Edge rows, 14 delivered, 2 exact
browser_binding_ambiguous refusals, 0 failed, and 0 skipped. The matrix
covers background typing, composed frames and capability-tested OOPIFs,
multi-tab exact binding, isolated preparation, stale refs, full roundtrip,
trusted background click, and same-bounds ambiguity. Every row includes a
fixture-state oracle, focus and z-order guards, no-leaked-input and cursor
checks, and a playable video.4fc4e00b0630f5889e17e91ebe31b3508dc3818f:
8 declared Chrome rows, 6 delivered, 2 refused, 0 failed, and 0 skipped.
Trusted pointer input refused with
browser_input_trust_unavailable before dispatch because standalone
Chromium activates its native window on that route. Same-bounds ambiguity
refused with browser_binding_ambiguous. Explicit DOM click, ref-bound
typing, navigation, composed frames, multi-tab binding, stale refs, and
isolated preparation passed with the complete background oracle set and
videos.input_route: "dom_event" is the explicit synthetic full-background click
route. The driver never changes from trusted input to a DOM event silently.Revision 32c77f775bd16d50507fbedb54a9682b9ab5599d passed the complete
browser-specific release matrix on the three available interactive cloud
desktops. Every declared result includes fixture-state and desktop-side-effect
oracles plus a playable video.
browser_route_unavailable.browser_input_trust_unavailable; the explicit DOM event
route remained available for synthetic full-background clicks. Electron
delivered and Tauri refused exactly.The broader native-Wayland shared-action diagnostic also found existing Tauri WebKitGTK pointer and scroll failures. Those rows do not use the browser CDP transport, and this branch does not change their injection paths. They remain separate platform-action gaps and are not counted as browser-tool acceptance.
The reusable macOS VM work was merged before the final replay. The retained Lume evidence below therefore validates the installed source in a disposable, permission-complete guest rather than relying on host TCC state.
The cold native-Sway replay after that setup correction passed all 8 standalone
Chromium rows: 6 delivered, 2 expected refusals, 0 failed, and 0 skipped. Each
retained environment.jsonl is the authoritative source identity ledger; the
documentation-only completion commit is replayed separately so the final
branch identity is not inferred from predecessor evidence.
| Environment | Standalone rows | Embedded rows | Accepted outcome |
|---|---|---|---|
| Windows interactive RDP | 16 | 3 | 19 passed: 15 delivered and 4 expected refusals; 19 playable videos |
| Linux X11 | 16 | 2 | 18 passed across Chrome, Snap Chromium, Electron, and Tauri; 18 playable videos |
| Linux native Wayland/Sway | 8 | 2 | 10 passed: 7 delivered and 3 expected refusals; 10 playable videos |
| macOS Lume | 8 | 3 | 11 browser rows passed; the complete canonical matrix also passed 148/148 with 149 playable videos including preflight |
The implementation candidate 19f1e09d55e2a595ccb17f1fcc48d65ceb5d181f
produced those accepted counts with zero skipped rows. One earlier macOS
canonical attempt missed the Electron foreground drag while the fixture stayed
idle; a fresh, uninterrupted worker replay passed that row and all 147 others
without retrying any action inside a result cell. The final documentation-only
branch head is validated again after this record is committed.
Safari/WKWebView/WebKitGTK mutation, Firefox BiDi mutation, split-process
WebView2 mutation, and mutation on Wayland compositors without exact pid and
geometry remain deliberate structured limitations. Existing user profiles are
never copied, restarted, or attached implicitly by browser_prepare.
A fresh exact-head Lume replay invalidated the earlier macOS completion claim. Electron completed all 40 canonical rows, but Tauri snapshots later exceeded the testkit's 25-second public call deadline. A second clean replay reached the same failure class at a different Tauri row, proving that this was not a single action assertion or fixture expectation.
The driver had wrapped the native AX walk in a 30-second Tokio timeout. The
public client therefore timed out first, and dropping the spawn_blocking
future could not cancel an AXUIElementCopyAttributeValue call already blocked
in ApplicationServices. The AX walker now applies Apple's finite native
messaging timeout to the application and every descendant AX object. The
driver-level walk deadline is also 20 seconds, below the public client limit,
so an unresponsive application returns a structured driver error instead of a
generic client timeout and cannot retain a worker indefinitely.
All prior Windows, Linux, and macOS artifacts remain diagnostic only until this correction is committed and every acceptance environment is replayed from that exact source revision. No test oracle, expected outcome, or declared row was weakened to accommodate the failure.
The finite AX deadline exposed separate fixture defects instead of masking
them. The canonical build scripts compiled a release Tauri executable without
the production custom-protocol feature, so the native window opened but its
embedded frontend was not served. The shared Tauri feature now maps to
tauri/custom-protocol, and both Unix and Windows fixture builders enable it.
The Tauri package also did not declare its copied ../web frontend directory
as a Cargo build input. A cached target could therefore retain an older page
after the shared fixture changed or a clean source snapshot was synced with
preserved mtimes. Its build script now emits rerun-if-changed for that
directory, making frontend embedding deterministic without deleting build
caches.
An official custom-protocol bundle exposed the second issue: starting the
first loopback fixture-state request before custom-scheme navigation finished
could keep the WKWebView load open. The shared web fixture now starts journal
publication from the window load event. It still publishes immediately after
that event and every 250 milliseconds thereafter, so the independent external
oracle is unchanged for actions. Electron, Tauri, WebView2, and WKWebView
continue to consume the same page; there is no macOS-only fixture or relaxed
readiness assertion. Exact-head cross-platform replay remains required before
release acceptance.
Further replay showed that the production custom scheme was not a reliable
fixture transport on the current macOS WebKit runtime. Tauri completed the
custom-scheme and data-URL navigation callbacks, but the page remained empty.
The fixture now starts a server on an ephemeral loopback port, serves the same
checked-in shared/web/index.html, and creates its only webview against that
exact URL. It does not contact an external service or duplicate the shared
page. Cargo also watches tauri.conf.json, so a cached build cannot retain an
older window declaration.
The final fixture passed all 40 Tauri action cells on a physical macOS host, covering foreground and background delivery across AX and PX addressing. Its embedded browser row also returned the expected side-effect-free route refusal. The independent fixture journal remained the action oracle.
The disposable macOS VM still renders both Tauri and the native WKWebView fixture as an empty surface. System logs show WebKit finishing the page load and then terminating an unresponsive GPU process. The same Tauri binary and page pass on the physical host, so this result is an environment limitation in that VM image rather than a browser-tool or fixture regression. The matrix keeps its readiness assertion; it does not convert the blank surface into a pass or an expected product refusal.
Implemented the reviewed existing-profile strategy without changing the driver-owned isolated profile contract:
browser_prepare accepts the tagged
strategy:{kind:"existing_profile"} request only with an exact pid,
native window, named session, and five-minute single-use artifact minted by
the interactive browser-approve CLI. The ordinary MCP host marker cannot
authorize this route.browser_input_incomplete for a partial prefix.The macOS Chrome adapter discovers a unique PID-owned loopback listener and uses a bounded native AX adapter. It acts only on one browser-owned modal sheet with remote-debugging corroboration and one semantic Allow button advertising AXPress. During physical-host validation, Chrome exposed the same top-level window through both AXChildren and AXWindows with different proxy pointers; top-level deduplication now uses Core Foundation object equality so a single sheet cannot become a false ambiguous match.
A fresh disposable Chrome profile provided the real acceptance evidence. The
user-facing remote-debugging toggle exposed one PID-owned listener; the exact
CLI artifact authorized one call; Cua Driver pressed Chrome's consent action;
browser_prepare returned attached_existing_profile; and the following
get_browser_state(pid, window_id, session) produced an exact
native_cdp_window binding without launching, restarting, copying, or changing
the profile. A separate disposable-profile standalone harness proves the same
public contract and a DOM mutation against an independent fixture oracle.
Final local verification passed 236 core tests, 134 macOS platform tests, 102
CLI tests, the protocol and session integration suites, and the ignored
source-built standalone_browser_existing_profile real-browser row. The shared
Windows/Linux/testkit packages compile with the additive contract. The exact
branch build was installed locally as Cua Driver 0.8.3; its stable app identity
retained both Accessibility and Screen Recording grants.
Windows and Linux compile the same contract, grant, generation, mutation, and refusal model. Their native consent adapters remain strict refusals until an interactive UIA or AT-SPI harness proves stable browser-owned prompt semantics; no unsupported prompt route is advertised as working.
The committed source snapshot was also replayed natively on isolated remote VM
staging paths. The interactive Windows VM passed all 236 shared-core tests and
96 Windows platform tests, with the two pre-existing cache UAF repro cases
remaining ignored. The Wayland VM passed all 237 Linux shared-core tests and
all 100 Linux platform tests. These are native contract and platform-crate
results, not evidence of UIA or AT-SPI consent-dialog automation. An additional
X11 VM attempt reached dependency compilation but its root filesystem had only
4 MB free and returned No space left on device; it produced no behavioral
verdict and was not counted as product evidence.
The existing-profile route now has product-specific setup descriptors for Chrome, Chromium, and Edge and native exact-control adapters on all advertised desktop platforms:
Linux Chromium-family processes used by this harness include
--force-renderer-accessibility; production calls without a complete AT-SPI
tree refuse with the missing prerequisite. Sway focus is restored through the
exact compositor container guard. Generic Wayland sessions without exact PID,
window, and geometry identity are not promoted into a mutation route.
All setup paths still require an operation-bound one-use approval artifact, prove a loopback listener owned by the approved PID, attach one browser-level socket, invalidate old capabilities, report every visible side effect, and attempt exact rollback after a failed setup. Safari, Firefox, unrecognized products/locales, and ambiguous setup or consent controls remain refusals.
The evidence in this entry was produced from a dirty diagnostic source marker while the implementation was converging. It establishes behavior but is not release acceptance; the final commit must be synced and replayed with its exact SHA on every retained platform lane.
The final acceptance replay used exact source SHA
7fa6c80f5c433a7d4dadef75e4873af89de0cd3f. A preceding Windows replay was
discarded because host input and another runner contaminated the interactive
desktop. A macOS replay then exposed a testkit posture defect: the foreground
sentinel waited for Electron to focus itself before the harness explicitly
activated it. The final testkit change waits for readiness, activates the exact
sentinel, and proves native focus plus the fixture focus journal. It does not
change any browser action or oracle.
| Platform | Representative products | Rows | Delivered | Expected refusals | Failed | Skipped | Playable videos |
|---|---|---|---|---|---|---|---|
| Windows/Win32 | Chrome, Edge | 20 | 18 | 2 | 0 | 0 | 20 |
| macOS/Quartz | Chrome, Edge | 20 | 16 | 4 | 0 | 0 | 20 |
| Linux/X11 | Chrome | 10 | 8 | 2 | 0 | 0 | 10 |
| Linux/Wayland | Chromium on Sway | 10 | 8 | 2 | 0 | 0 | 10 |
| Total | 60 | 50 | 10 | 0 | 0 | 60 |
All 60 declarations ran. Every result had test_status=pass; every MP4 was
downloaded and independently checked for positive duration. The Windows lane
ran in a live RDP user desktop rather than Session 0. The Linux lanes used
Openbox/X11 and native Sway/Wayland respectively, while macOS ran in a
TCC-authorized disposable Lume guest.
The expected refusals are product signal, not environment skips. Every platform
refused browser_binding_ambiguous when two native browser windows had the same
bounds and no unique title tie-break. macOS and Linux additionally refused
browser_input_trust_unavailable for Chromium's trusted CDP Input click because
that route activated the standalone target; the ref-targeted dom_event route
remains the synthetic full-background alternative. Windows delivered trusted
clicks for both Chrome and Edge without changing the sentinel posture.
The Sway wrapper replaced itself with nix develop, so its redundant shell EXIT
trap could not write an exit-marker file. The canonical runner nevertheless
finished, produced its summary, exited from the process table, recorded all 10
passing rows, and generated 10 valid videos. The acceptance verdict is based on
those canonical artifacts rather than the disposable wrapper marker.
Added the opt-in semantic_v2 snapshot contract while retaining
dom_refs_v1 as the compatibility default. The new collector joins the
Chromium accessibility tree, pierced author DOM, layout snapshot, viewport
metrics, and proven frame identities. It separates compact readable content
from typed action refs, ranks visible state before offscreen state, excludes
CSS-hidden retained controls before budgeting, and conservatively identifies
controls covered by fixed or absolute page overlays.
Semantic action refs declare click and/or type. Mutation tools now return
browser_action_unavailable before delivery when a semantic ref does not
declare the requested action. Static nodes with live backend identity are
returned separately as content_refs for read-only subtree scope; they do not
become clickable. A bounded DOM supplement recovers visible custom controls
with explicit interaction evidence when AX omits them.
Read scoping supports role/name/text query, current scope_ref, and opaque
single-use continuation. Continuations remain inside the session capability
store and are invalidated by use, a newer snapshot, navigation, reconnect,
target replacement, or session end. Raw CDP target IDs, backend node IDs,
object IDs, selectors, and continuation offsets remain private.
Known full-DOM size or serialization failures use a progressively shallower
depth ladder and hydrate truncated branches with bounded DOM.describeNode
calls. Time, call-count, and scan-node budgets terminate hydration and retain
complete:false; unrelated transport failures remain hard errors. The
deterministic fixture contains 320 CSS-hidden retained controls, visible
message/editor/actions, and more than 300 offscreen controls. Focused tests
prove visible-first state, continuation, query, subtree scope, action-kind
enforcement, modal occlusion, DOM supplementation, bounded fallback and
hydration, and stale continuations after both newer snapshots and out-of-band
navigation. Hidden-node omission counts are deduplicated, and icon-font glyphs
are removed from semantic text.
Local source validation passed the complete core and session-lifecycle suites, all 46 testkit library tests, source compilation of the standalone browser harness, generated documentation drift checks, public-doc hygiene, and local link checks. A source-installed macOS product-path replay passed the real Chrome semantic row with the TCC-authorized daemon, foreground sentinel, fixture journal, exact DOM click, re-snapshot, and typed-value oracle. Windows and Linux X11 exact-SHA workflow evidence is the remaining release-acceptance gate.
The bind response no longer labels a representative DevTools target as the
active native tab. A unique exact native-title match produces one true and
the remaining tabs false; duplicate, empty, substring-only, or otherwise
ambiguous titles produce null for every tab. The standalone harness adds a
same-title adversarial row so future list-order regressions are externally
visible.
The typed surface now includes page-owned JavaScript dialog inspection and
resolution, exact file-input assignment, extended pointer actions, and
approval-scoped downloads. All mutations retain the existing exact binding,
tab lock, reconnect generation, ref/frame identity, and fail-closed posture.
Synthetic pointer actions are explicit dom_event requests; trusted pointer
actions never fall back after a refusal. Downloads serialize Chromium's
browser-wide setting, subscribe before triggering, correlate exact frame and
opaque id events, restore defaults on every outcome, and remove only a proven
direct partial file after failure.
The real-browser harness adds separate rows for dialogs, upload, hover/right-click/double-click/scroll/drag, and downloads. A fixed loopback attachment keeps download evidence deterministic. Background rows use the foreground sentinel and fixture-owned state rather than treating a CDP reply as delivery. Linux Chromium cannot resolve a native JavaScript modal without changing foreground posture, so its matrix records a no-dispatch background refusal and a separate explicit foreground delivery row. Windows and macOS retain strict background dialog resolution. Trajectory recording redacts prompt text, upload paths, download directories, and internal approval markers; product telemetry retains only fixed operation and refusal categories.
The dialog design adapts the exact-target, no-replay security model introduced
in PR #2166. Its contributor remains credited through a parsed
Co-authored-by trailer in the landing history and Salvaged from #2166 in
the pull request body even though this implementation supersedes that earlier
code shape. Issue authors remain reporter-attributed when the landing PR
closes their linked issues.