cua-driver — Windows

Orchestrates Windows app automation via the qwen-cua-driver binary (qwen-cua-driver.exe). Whenever a user asks to drive a native Windows app, follow the loop in this doc rather than calling tools ad-hoc — the snapshot-before-action invariant is not optional and silently breaks if you skip it.

SKILL.md in this directory describes the cross-platform core; this file is the Windows-specific extension. Read both: the snapshot invariant, MCP-vs-CLI choice, agent cursor overlay, and recording flow are identical. The launch, click, and accessibility- tree mechanics in this file replace the macOS ones.

The no-foreground contract — read this first

The user's frontmost app MUST NOT change. Users pay for the right to keep typing in their editor while an agent drives another app in the background. Violate this rule and every other nice property the driver gives you (no cursor warp, no taskbar flash, no window restore-and-raise) stops mattering — you just shipped a SendInput wrapper with extra steps.

How the contract is enforced per call: the delivery_mode field

Every Windows input tool (click, double_click, right_click, drag, scroll, press_key, hotkey, type_text) accepts an optional delivery_mode field — this mirrors the macOS delivery_mode surface (same name, same two values). The default is "background" — strict no-foreground:

macOS is the source of truth — background never alters the screen. Earlier Windows builds "cheated" in background with three tricks that this pass removed: (1) a z-order raise (ZorderGuard) to win the pointer hit-test on occluded windows, (2) a full focus-activate for WPF drags, and (3) a cloaked (hidden) focus-grab for keystrokes/text the target would otherwise drop. macOS does none of these (pure CGEvent-to-pid + focus suppression), so Windows now does none either: when strict no-front / no-raise delivery can't land, the tool returns background_unavailable and the agent — not the driver — decides whether to escalate to delivery_mode:"foreground" (the rung where fronting is explicitly opted into). A background call will never raise, restack, flash, or steal focus.

Removed: the legacy "auto" mode. Earlier builds had a third Windows-only dispatch:"auto" mode (silent SendInput fallback on known-problematic targets). It was removed in the macOS-alignment pass because it could front the target without the caller opting in — breaking the no-foreground contract macOS guarantees. Any unrecognised value (including a stray "auto") now resolves to "background". If you have notes/snippets that pass dispatch:"auto", switch to an explicit delivery_mode:"foreground" for the cases that need fronting.

Always try delivery_mode:"background" first

The hit-test fallback above means delivery_mode:"background" is the correct default even when the target is a XAML host whose input stack drops raw PostMessage (UWP Calculator, Win11 Notepad, WinUI3 apps, etc.). cua-driver turns the pixel coord into a UIA Invoke at that point and delivers through the accessibility channel — no flash, no focus steal. Only escalate to delivery_mode:"foreground" when you actually see a background_unavailable structured error. Retry only that action with delivery_mode:"foreground"; cua-driver briefly activates the target, delivers the input, and restores the previous foreground.

Empirical: pixel-clicks via delivery_mode:"background" against the UWP Calculator on Win11 (Number-pad buttons + operators) consistently resolve through try_invoke_in_window_at_point and produce "✅ Performed UIA Invoke at (sx,sy) for pid X." with zero visible flash. delivery_mode:"foreground" on the same coords also works but flashes the Calculator window foreground for ~40 ms — it's the costlier path; only use it for surfaces with no UIA peer.

background_unavailable error shape:

{
  "isError": true,
  "structuredContent": {
    "code": "background_unavailable",
    "target_class": "Chrome_WidgetWin_1",
    "event_kind": "mouse_click",
    "escalation": { "recommended": "foreground", "reason": "occluded / known-dropped event kind" },
    "suggestion": "Retry this action with delivery_mode:\"foreground\"."
  }
}

Errors retain their diagnostic escalation.recommended hint. Successful action results use the narrower escalation.target contract instead (see SKILL.md → behavior matrix). On Windows this error recommendation is "foreground" because the dropped event needs the fronting rung. (Contrast macOS / X11, where a background pixel click can still land in the background.)

The normal flow when an agent gets that error:

1. Reissue only the refused action with delivery_mode:"foreground".
2. cua-driver activates the target, delivers through SendInput, and restores the previous foreground.
3. Continue with delivery_mode:"background" for later actions unless they are also refused.

Persistent focus-proxy exception

bring_to_front is not part of the normal input ladder. Use it only when a focus-proxy surface must remain foreground across multiple calls, such as an RDP or Windows App session, or when repeated action-scoped activation prevents the remote surface from accepting input.

The bring_to_front tool uses an AttachThreadInput trick to attempt the foreground swap even when the daemon isn't at UIAccess integrity (the same trick that powers send_key_synthesized). Returns {previous_fg_hwnd, now_fg_hwnd, landed_on_target} — check landed_on_target. Without UIAccess, Windows' foreground-lock can still reject the swap (and a subsequent delivery_mode:"foreground" call will bail with the "Foreground swap … was rejected by Windows" diagnostic rather than landing input on the wrong window). When that happens the target genuinely cannot be driven by SendInput/keystrokes in this session: use an interactively launched High-IL daemon. The reserved cua-driver-uia worker is a daemon-internal, default-off service boundary and public clients must never connect to its pipe directly. Alternatively, for tasks that produce a file — generate the document and launch_app it instead of driving the GUI (e.g. building a spreadsheet and opening it in LibreOffice Calc rather than typing into the grid, which is dropped on the VCL background path).

Before running any shell command, ask: "does this raise, activate, foreground, or steal focus from any app?" If yes, don't run it. Every one of the commands below activates the target on Windows and is therefore forbidden unless the user explicitly asked for frontmost state:

• Start-Process <exe> / Start-Process <url> / Start-Process -FilePath ... — defaults to launching with SW_SHOWNORMAL which activates the new window. Windows treats new processes as user-initiated foreground apps. The CmdLine flag -WindowStyle Hidden helps but does not block activation for apps that call SetForegroundWindow themselves on startup (Edge, most browsers, Office, most installers). Never use Start-Process to launch a visible app. Use launch_app — it goes through SW_SHOWNOACTIVATE and wraps an internal VK_NONAME keybd_event trick that the OS treats as "user activity from another window," so the target's own SetForegroundWindow call is denied per Windows focus-stealing prevention rules.

• & "C:\path\to\app.exe" from PowerShell — same as Start-Process on activation. PowerShell's & invocation operator spawns the process with foreground intent.

• cmd /c start <thing> and start /b <thing> — start on Windows is the equivalent of macOS's open: it goes through the shell's protocol-handler / file-association lookup and activates the resulting process. start /min helps for taskbar minimization but still activates the new window before minimizing it (flash visible to the user). Forbidden for the same reason.

• explorer.exe shell:AppsFolder\<AUMID> / explorer.exe ms-edge: <url> — these are the Windows-shell equivalents of open -a / open <url> on macOS. They go through IApplicationActivationManager with the wrong activation kind and foreground the target. Use launch_app({aumid}) or launch_app({urls}) instead — those route through IApplicationActivationManager::ActivateApplication with the correct flag combination that respects SW_SHOWNOACTIVATE.

• SetForegroundWindow(hwnd) / BringWindowToTop(hwnd) / SwitchToThisWindow(hwnd, fAltTab) / ShowWindow(hwnd, SW_SHOW) / SetActiveWindow(hwnd) — all foreground the named window by definition. Never call these from Bash via PowerShell add-types or C# inline. If you find yourself doing this to "make a click land," the click is already wrong: re-read "Click semantics" below.

• AttachThreadInput + SetForegroundWindow trickery — the classic Windows focus-bypass hack. Even when it works it's a visible focus pop and a UIPI/UIAccess violation. Forbidden.

• SendInput(MOUSEEVENTF_*) over the user's real cursor — moves the cursor and synthesizes input. The cursor warps to coordinates, any handler in the topmost window at those coordinates fires, and for most apps the receiving window activates because input arrived from the OS-trusted pipeline. Use click({pid, x, y}) or click({pid, element_index}) instead — both route per-pid and never touch the OS cursor.

• SendInput(KEYBDINPUT) with no target HWND — same idea: goes to the focused window, not your target. Use hotkey({pid, keys: [...]}) which uses PostMessage(WM_KEYDOWN/UP) to the named pid's focused window.

• Keyboard shortcuts that semantically mean "focus here" — Chromium / Edge / Firefox Ctrl+L (focus address bar), Explorer's Ctrl+L / Alt+D (focus path bar), F6 (focus cycle). These aren't pure key events — the receiving app interprets "user wants to type here" as activation intent and raises its window to be key. Even when delivered to a backgrounded pid via hotkey, the downstream app pulls focus. For omnibox navigation specifically, the correct path is launch_app({path: "...msedge.exe", urls: ["https://…"]}) (or {aumid: "Microsoft.MicrosoftEdge.Stable_…!App", urls: [...]}) — no omnibox dance, no Ctrl+L, no focus-steal. The browser opens the URL in a new window without activating it.

• Tab-switching shortcuts in browsers (Ctrl+1..9, Ctrl+Tab, Ctrl+Shift+Tab, Ctrl+PageUp/Down) are visibly disruptive even when delivered to a backgrounded pid. The app's key handler processes the shortcut, the window re-renders the new tab's content, the user sees their tabs flipping. Same as macOS: there is no UIA-only workaround — page content (HTML, DOM, WebView2's accessible tree) populates only for the focused tab; inspecting a background tab requires activating it, which is the visible flip.

  Prefer the windows-over-tabs pattern: for each URL you need to drive backgrounded, use launch_app({urls: [url]}) — Chromium- family browsers open each URL in a new window. Each window has its own window_id, its own UIA tree, and can be inspected / interacted with via element_index without activating or switching anything. Tabs are a UX grouping for humans; cua-driver-rs workflows should default to windows.

• Win+key shortcuts owned by the shell — Win+E (Explorer), Win+R (Run), Win+S / Win+Q (Search), Win+number (taskbar pinned-app activation), Win+Tab (Task View), Alt+Tab (window switcher), Win+D (show desktop), Win+L (lock), Win+M / Win+Up/Down (minimize / maximize). All hard-coded to the shell and visibly disruptive — they bypass any per-pid routing. Forbidden in hotkey calls regardless of target pid.

• taskkill /F /IM <exe> to close an app the user is using — not a focus-steal but a data-loss vector. Use hotkey({pid, keys:["alt","f4"]}) and let the app close cleanly, or ask the user first.

Reading state is fine. Listing windows, reading registry, querying process info via Get-Process, calling tasklist, walking UIA trees via qwen-cua-driver get_window_state — none of these change focus. Mutating state via shell shims is the line.

Corollary — the Win+Search rule. Don't use Win+S/Win+Q "open Start search, type query, press Enter" patterns to launch apps. They (a) foreground the Search UI, (b) leave the Search UI populated with the agent's typed text, (c) trigger the new app via the Start Menu's own activation path which SW_SHOWNORMALs it. Use launch_app({name}) / {aumid} / {path} instead.

"Open <app>" in user speech means launch, not activate. qwen-cua-driver launch_app is the one correct path for process startup — it's idempotent (no-op on a running app), returns the pid, and internally uses SW_SHOWNOACTIVATE plus the AppX-broker activation flow for packaged apps so the target's window comes up without becoming foreground. The macOS FocusRestoreGuard is replaced on Windows by the activation-deny dance: the launcher pumps a VK_NONAME keystroke before activating, which Windows interprets as "another app just had user activity" and rejects the target's own SetForegroundWindow call per Windows focus-stealing prevention rules.

launch_app foreground-restore behaviour (Windows). In addition to SW_SHOWNOACTIVATE, launch_app captures GetForegroundWindow() before the spawn and schedules a post-spawn polling restore (every 100 ms for up to 3 s) that flips the foreground back to the prior window once the spawned app has actually activated. This mirrors the macOS FocusRestoreGuard. For urls-only invocations (open these links in the default browser, no app-identifying field) the restore is skipped, because the user explicitly asked for the page to come up. The restore is best-effort: SetForegroundWindow from non-UIAccess processes is subject to Windows' foreground-lock and may silently no-op — failures are logged at tracing::trace! and not surfaced as errors.

hotkey modifier dispatch on Chromium/Electron (Windows). When the target is detected as a Chromium-family window (Chrome_WidgetWin_* class — covers every Chromium browser AND every Electron app: Chrome, Edge, Brave, Arc, Vivaldi, Slack, VS Code, Discord, Teams, Notion), hotkey with modifiers routes through PostMessage(WM_KEYDOWN/UP) instead of SendInput — no foreground swap. Chromium's Browser::HandleKeyboardEvent reads modifier state from the WM_KEYDOWN LPARAM bits, NOT from GetKeyState, so PostMessage delivery works for real accelerators (Ctrl+T, Ctrl+W, Ctrl+L, Ctrl+Shift+B, etc.). The SendInput-swap path (send_key_synthesized) remains the dispatch for non-Chromium Win32 + modifiers — classic apps (LibreOffice, FAR, classic Notepad) use TranslateAccelerator which requires the system modifier state updated, and PostMessage can't do that.

modifier on a background click is a Windows residual. A backgrounded click delivers through UIA Invoke or PostMessage, and neither carries live keyboard state — so a modifier (Ctrl/Shift/etc.) passed alongside a delivery_mode:"background" click is not honored on Windows. The modifier parameter is part of the shared schema and is accepted everywhere; it only takes effect on the SendInput rung, i.e. a delivery_mode:"foreground" click, where SendInput sets real modifier state. If you need a modifier-click on Windows, escalate that one action to foreground. Reserve bring_to_front for the persistent focus-proxy exception described above.

Cross-platform schema residuals (Windows)

The capture/dispatch/addressing params are a shared cross-platform contract (see SKILL.md → Cross-platform parameter contract). Three Windows-relevant notes:

• session is now accepted on every action/cursor tool. Earlier Windows builds rejected session via additionalProperties:false (it was effectively a macOS-only key); the shared contract makes it uniformly schema-accepted — cursor-wired where a cursor glides, accepted elsewhere.
• debug_window_info is a Windows-only tool (window-handle / class / rect / z-order diagnostics for triaging the click chain). It is deliberately not part of the cross-platform surface — there is no macOS or Linux counterpart.
• launch_app identifiers are platform-specific. Windows takes aumid / launch_path / path / start_minimized (plus bundle_id overloaded for AUMIDs); macOS takes bundle_id / urls. name is the portable fallback. See the AUMID section below.

Chromium pixel-click foreground polling restore. click({pid, x, y}) on a Chromium target falls through to send_click_synthesized (SendInput + brief foreground swap) because Chromium's input thread filters by queue-origin and PostMessage-delivered clicks don't fire DOM events. The synchronous restore inside send_click_synthesized covers the immediate swap; an additional polling guard (same shape as launch_app's FocusRestoreGuard) catches the asynchronous Chromium re-activation that can happen as the renderer's input handler processes the click (focus().activate() / WebContents::Activate() — 100-500 ms later). The guard is gated on GetWindowThreadProcessId(fg_now) == pid so user Alt-Tabs are respected. The polling guard is asynchronous and best-effort, so the tool response is not proof that the previous foreground has already been restored.

Defaults — always prefer cua-driver over shell shims

Default transport is the qwen-cua-driver CLI — Bash shelling out to qwen-cua-driver <tool-name> with JSON piped via stdin (avoids PowerShell 5.1's argv quoting quirks for strings containing both quotes and spaces). MCP tools (prefix mcp__cua-driver__*) only when the user explicitly asks for them. CLI wins because it picks up rebuilds instantly, failures are easier to diagnose, and there's no per-tool schema-load overhead.

Every reference to click(...), get_window_state(...) etc. in this doc means qwen-cua-driver <name> with JSON piped via stdin — translate to MCP form only when MCP is requested.

CLI argument plumbing on Windows

Three equivalent shapes for passing JSON to qwen-cua-driver <tool>:

1. Stdin pipe (recommended) — avoids PS quoting bugs entirely:
   powershellCopy

   '{"pid":1234,"text":"hello world"}' | & qwen-cua-driver call type_text
   

2. Positional with escaped quotes — works for JSON without spaces in string values:
   powershellCopy

   & qwen-cua-driver call list_windows '{\"app_name\":\"Calculator\"}'
   

   (Windows PowerShell 5.1 mangles {"x":"with space"} when both " and appear unquoted in argv. Use stdin for those.)
3. --% stop-parser directive (PowerShell 5.1 specific):
   powershellCopy

   & qwen-cua-driver call type_text --% {"pid":1234,"text":"hello world"}
   

Stdin is the only path immune to all PS quoting edge cases. Prefer it.

Intent → tool mapping

If you find yourself reaching for the right column, something has gone wrong — re-read "The no-foreground contract" above.

The narrow carve-out

The only legitimate use of SetForegroundWindow or Start-Process with a foreground app is when the user explicitly asked for frontmost state ("bring Edge to the front", "make Calculator visible", "I want to see it"). Reaching for it because a tool call returned something confusing is wrong — diagnose first.

When a qwen-cua-driver call surprises you, diagnose qwen-cua-driver first:

• Posted click to pid X instead of Performed UIA Invoke ...? The (x,y) UIA hit-test didn't find an InvokePattern-bearing element at that pixel inside the target HWND, so it fell through to PostMessage(WM_LBUTTONDOWN). For UWP / WebView2 surfaces, PostMessage silently no-ops — re-snapshot via get_window_state(pid, window_id) and use element_index so the daemon can invoke the cached UIA element by identity instead of by point.
• Invalid element_index / No cached UIA state? You either skipped get_window_state this turn or passed a different window_id than the one the snapshot cached against. The cache is keyed on (pid, window_id) — indices don't carry across windows of the same app. Re-snapshot with the same window_id you're about to click in.
• Invalid window handle (0x80070578)? The HWND you passed is stale (window closed, recreated). Re-resolve via list_windows.
• Empty tree_markdown / sparse UIA tree? Some apps populate their UIA tree lazily on first call; retry get_window_state once. If still empty, the app has no UIA provider — fall back to an element px action (x,y clicks off the screenshot) on visible content (acceptable for exploration; pair with screenshots).
• Empty tree, or a snapshot with no image? get_window_state returns both the UIA tree and a screenshot by default — there is no capture mode to pick. If the tree came back empty, the response is degraded (no UIA provider — retry once for lazy trees, see the note above); act by px off the screenshot in the same response. The capture_mode param is deprecated and ignored — it's still accepted so old callers don't error, but both the tree and the image come back regardless of what you pass.
• get_desktop_state returns desktop_scope_disabled? That's intended: full-display capture is a desktop-scope operation, gated by the caller-declared session policy. To verify a specific window use get_window_state(pid, window_id) (works backgrounded). Use get_desktop_state only in a strict desktop session or after explicitly escalating an auto session once the window ladder is exhausted. Desktop actions pass scope:"desktop" with no pid/window_id. Don't reach for get_desktop_state as a casual screenshot — it's the capture surface for desktop-scope coordinate loops, not window inspection.
• Calc display stuck at 0 after my clicks? Almost always means UWP and you're on the PostMessage path. UWP processes pointer input via Windows.UI.Input, NOT through HWND message queues — PostMessage(WM_LBUTTONDOWN) gets ignored. Use element_index instead of (x,y) for UWP targets.

Only after those are ruled out should you fall through to the activate fallback. Always name the focus steal in your response ("I'll briefly bring Edge to the front because …").

Self-check pattern

Before every Bash call whose command line touches any Windows app (launching, opening, clicking, typing, scripting, screenshotting), run the self-check:

1. Does this command foreground the target? If yes — stop and translate to the cua-driver equivalent from the mapping table.
2. Does this command move the user's real cursor? (SendInput, SetCursorPos from inline C#, AutoHotkey scripts, nircmd sendmouse.) If yes — stop; use click({pid, x, y}) which routes per-HWND via PostMessage / per-element via UIA Invoke and never warps the cursor.
3. Does this command bypass cua-driver entirely? (PowerShell GUI scripts, AutoHotkey, SendKeys, AppleScript-equivalent automation tools.) If yes — stop; find the cua-driver tool that does the intent.

If all three are "no," the command is safe. If you can't answer, default to stop and ask rather than proceed. A single Start-Process run by accident steals the user's foreground and kills the trust your prior tool calls earned.

Prerequisites — check before starting

1. qwen-cua-driver is on $PATH — Get-Command qwen-cua-driver or where.exe qwen-cua-driver. Install location: %LOCALAPPDATA%\Programs\Qwen\qwen-cua-driver\bin\qwen-cua-driver.exe, added to the user PATH by the install script. If missing, point the user at:
   powershellCopy

   irm https://raw.githubusercontent.com/QwenLM/qwen-code/main/packages/cua-driver/scripts/install.ps1 | iex
   

   and stop.
2. The runtime owner must run in an interactive session (Session 1+), NOT Session 0. This is the daemon for one-shot CLI/service mode and the MCP process for bare stdio MCP. Windows isolates services into Session 0 with no desktop. UIA enumeration, screenshot via PrintWindow, and IApplicationActivationManager all silently return empty / timeout in Session 0. Check:
   powershellCopy

   Get-Process qwen-cua-driver | Select Id,SessionId
   

   SessionId == 0 is refused before runtime actions. The autostart Scheduled Task uses LogonType=Interactive so the daemon lands in the user's logon session. If you started the daemon via SSH-into-Windows, that session is usually Session 0 — kick the autostart task instead:
   powershellCopy

   schtasks /Run /TN qwen-cua-driver-serve
   

3. Run qwen-cua-driver doctor — reports session ID, COM apartment status, UIA reachability, install paths, version. If anything reads false / error, fix that before tool-calling.
4. Permissions — Windows has no TCC equivalent. cua-driver-rs needs:
   • No admin elevation for normal use (UIA, PostMessage, UWP activation all work from a standard user token).
   • UAC elevation only for autostart registration if a system-wide scheduled task is requested (per-user task is non-elevated and the default).
   • SmartScreen: on a fresh install, Windows Defender SmartScreen may flag the unsigned binary on first run. Click "More info → Run anyway" once.

Using cua-driver from the shell

Tool names are snake_case, management subcommands are kebab-case — no ambiguity. Tools invoked as qwen-cua-driver call <tool-name> with JSON via stdin or positional arg. Management subcommands:

• qwen-cua-driver serve — start the persistent daemon used by one-shot CLI calls or by MCP clients that explicitly select it with --socket. Bare qwen-cua-driver mcp owns its runtime directly on Windows. Normally not run manually — the autostart Scheduled Task fires it at every interactive logon. If you stopped it (Stop-Process), re-run with schtasks /Run /TN qwen-cua-driver-serve, not by spawning qwen-cua-driver serve from SSH (Session 0 problem).
• qwen-cua-driver stop / status — daemon lifecycle.
• qwen-cua-driver doctor — full diagnostics.
• qwen-cua-driver list-tools / describe <tool> — tool surface discovery.
• qwen-cua-driver autostart {enable|disable|status|kick} — manage the Scheduled Task that auto-starts the daemon at logon. enable registers it (idempotent — replaces existing). kick runs it immediately without waiting for a fresh logon.
• qwen-cua-driver recording start|stop|status — see RECORDING.md. Windows video uses ffmpeg with gdigrab; trajectory evidence continues without video when ffmpeg is unavailable.

Over SSH, never use bare qwen-cua-driver mcp: the direct runtime rejects Session 0. Start the daemon in the interactive user session and run qwen-cua-driver mcp --socket \\.\pipe\qwen-cua-driver from SSH.

Canonical multi-step workflow:

# Daemon is already running via Scheduled Task.
# Launch UWP Calculator without focus-stealing.
'{"aumid":"Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"}' | & qwen-cua-driver call launch_app
# → {pid: 6004, windows: [{window_id: 459672, ...}]}

# Snapshot the UIA tree.
'{"pid":6004,"window_id":459672}' | & qwen-cua-driver call get_window_state
# Returns: tree_markdown with [N] indices plus structured element_token values,
# snapshot_id, screenshot, and dimensions.

# Click the "Equals" row with its opaque token from that response.
'{"pid":6004,"element_token":"s0000002a:22"}' | & qwen-cua-driver call click
# → "✅ Performed UIA Invoke on [22] ..."

# Re-snapshot to verify the action landed.
'{"pid":6004,"window_id":459672}' | & qwen-cua-driver call get_window_state

The core invariant — snapshot before AND after every action

Every action MUST be bracketed by get_window_state(pid, window_id):

• Before — the pre-action snapshot resolves the element_token you're about to use. A bare integer is rejected in 0.17; clients that keep integers must send the same response's snapshot_id. Targets from previous turns are stale; the server replaces the element index map on every snapshot, keyed on (pid, window_id). Indices from turn N don't resolve in turn N+1, and targets from window A don't resolve against window B of the same app. Skip this and element-indexed actions fail with stale_element_token or snapshot_id_required.
• After — the post-action snapshot verifies the action actually landed. Without it you can't tell a silent no-op from a real effect. The UIA tree change (new value, new window, disappeared menu, disabled button, etc.) is your evidence that the action registered. Especially important on Windows because the layered click path can return effect:"unverifiable" after PostMessage even when the click did nothing (UWP silently no-ops): the action result reports the route, not the task outcome. Only the re-snapshot tells you if the state changed.

Click semantics on Windows

Two click addressing modes, both gated by pid:

Snapshot-bound element mode (preferred)

{ "pid": 6004, "element_token": "s0000002a:22" }

Looks up the exact cached UIA element from the named snapshot, fires IUIAutomationInvokePattern::Invoke() on it directly.

Properties:

• No mouse cursor moves. The click is a UIA RPC, not an input event. The user's cursor stays where it is.
• No window activates. UIA Invoke does not foreground the target.
• Z-order is irrelevant. Works on backgrounded, occluded, minimized-but-not-iconic, and cross-virtual-desktop windows.
• Cross-process boundaries work. UIA marshals across the ApplicationFrameHost.exe → inner UWP process boundary (CalculatorApp.exe, Notepad-Win11.exe, etc.) — this is how it can click Win11 Calc buttons even though they live in a different process from the AppFrame HWND.
• Falls back to PostMessage(WM_LBUTTONDOWN/UP) to the deepest child HWND when the cached element doesn't expose InvokePattern (most edit fields, custom-drawn widgets, non-actionable elements). The fallback works for plain Win32 but silently no-ops on UWP. Read the closed action route: accessibility means UIA/MSAA and synthetic_events means the targeted event fallback. Do not parse the human-readable text.

This is the right path for any "click button N" or "click checkbox Y" intent. For a known application-menu hierarchy, prefer invoke_menu:

{ "pid": 6004, "window_id": 459672, "path": ["Window", "Arrange", "Left"] }

It uses ExpandCollapsePattern at intermediate hops and InvokePattern or SelectionItemPattern at the leaf, resolving the live UIA hierarchy again after every expansion. It refuses ambiguous, missing, or disabled segments and never falls back to pixels. Verify the command's semantic effect afterward.

(x, y) mode (element px action / pixel)

{ "pid": 6004, "window_id": 459672, "x": 446, "y": 671 }

Window-client coordinates (origin at the top-left of the screenshot the agent saw). The driver:

1. Converts to screen coords via ClientToScreen(hwnd, ...).
2. UIA hit-test in target HWND's subtree — ElementFromHandle resolves the root, FindAll(TreeScope_Subtree) enumerates descendants, picks the smallest-area InvokePattern-bearing element whose CurrentBoundingRectangle contains the screen point, calls Invoke(). This is the only path that lands on UWP / WebView2 / DirectComposition surfaces.
3. PostMessage fallback — if step 2 returned false (no Invokable element under the pixel inside hwnd, or hwnd has no useful UIA tree at all), fires PostMessage(WM_LBUTTONDOWN/UP) against the deepest child HWND at the screen point. Covers plain Win32 native controls.

Properties:

• No real cursor movement. The agent overlay glides + pulses for visual confirmation; the OS cursor is untouched.
• No focus steal. Both UIA Invoke and PostMessage are async per- pid / per-element; the target's window does not activate.
• Z-order independent. UIA hit-test honors the HWND-rooted subtree regardless of what's covering it on screen. PostMessage goes directly to the message queue.

Use this when the agent doesn't have a UIA snapshot in scope (zero- shot from a screenshot), or when the element it wants doesn't appear in the UIA tree (custom-drawn elements, canvas content, browser DOM nodes inside a WebView2 viewport).

What (x, y) mode does NOT solve

Apps with no useful UIA tree AND that ignore WM_LBUTTONDOWN on the HWND queue — primarily DirectX / OpenGL / Vulkan-rendered surfaces (games, custom renderers). The click chain falls all the way through and the click no-ops. For those, the only options are:

• Bring the window to top first (focus steal — ask the user before doing this, and document why), then synthesize input
• Use the app's keyboard interface via hotkey if available

SendInput is not a silent-fallback option here — it would steal focus from whatever the user is doing.

Right-click and multi-click

button: "right" and count > 1 skip the UIA Invoke step and go directly through the PostMessage path. Reason: UIA has no clean by-coord equivalent of ShowContextMenu, and Invoke() is single- fire by definition. The action result reports route:"synthetic_events" regardless of the target's UWP-ness — this is expected and will not work for UWP context menus. To open a UWP context menu, prefer hotkey({pid, keys: ["shift", "f10"]}) against the focused UWP element.

UWP / packaged apps — the AUMID layer

Modern Win11 apps (Calculator, Notepad, Settings, Photos, Edge UI chrome, Microsoft Store) are packaged apps with an App User Model ID (AUMID) rather than a plain .exe path. The AUMID looks like Microsoft.WindowsCalculator_8wekyb3d8bbwe!App — package family name + ! + AppId.

Windows architectural quirks that matter:

1. The .exe in C:\Windows\System32\notepad.exe is a 7 KB stub that exits immediately on Win11. It exists for backward compatibility but the real Notepad lives in the Microsoft.WindowsNotepad AppX package. Start-Process notepad spawns the stub, which exits, which redirects through the AppX broker, which spawns the real process with a different pid. You end up holding a pid that's already gone. launch_app handles this transparently — it detects AUMID-looking strings and routes through IApplicationActivationManager::ActivateApplication, which returns the real packaged-process pid.
2. UWP windows are hosted by ApplicationFrameHost.exe. The outer top-level HWND (the one with the title bar, the one EnumWindows enumerates) is owned by ApplicationFrameHost.exe, pid varies, not the same as the UWP's own pid. The actual UWP content runs in a separate process (e.g. CalculatorApp.exe). list_windows reports the AppFrame's HWND because that's what GetWindowRect, PostMessage, BitBlt all target. UIA transparently crosses the boundary into the inner process when you walk the tree.
3. AUMID resolution by name — launch_app({name: "calc"}) will first try a shell:AppsFolder lookup, matching against installed-package display names. On a hit it goes through the packaged-app path (real pid). Otherwise it falls back to ShellExecuteEx's PATH search (which hits the stub .exe). Prefer explicit aumid when you know it.

Known AUMIDs for common Win11 apps

Microsoft.WindowsCalculator_8wekyb3d8bbwe!App
Microsoft.WindowsNotepad_8wekyb3d8bbwe!App
Microsoft.MicrosoftEdge.Stable_8wekyb3d8bbwe!App
Microsoft.WindowsTerminal_8wekyb3d8bbwe!App
Microsoft.Paint_8wekyb3d8bbwe!App
Microsoft.WindowsCamera_8wekyb3d8bbwe!App
Microsoft.WindowsAlarms_8wekyb3d8bbwe!App
Microsoft.MicrosoftStickyNotes_8wekyb3d8bbwe!App
windows.immersivecontrolpanel_cw5n1h2txyewy!Microsoft.Windows.ImmersiveControlPanel  # Settings

To find an AUMID at runtime:

Get-StartApps | Where-Object Name -like "*Calculator*"

Browsers and embedded webviews on Windows

Use the typed, exact-binding workflow in BROWSER.md for Chrome and Edge page content. Windows has validated trusted background browser clicks when the driver runs in an interactive user desktop. A daemon in Session 0 cannot provide representative UIA, capture, focus, or browser evidence.

Ref- and coordinate-targeted browser mutations also drive the declared session's agent cursor. The adapter converts the live CDP page point into the bound window's DPI-aware screen coordinates, pins the overlay above that window, and keeps only the selected tab's session cursor visible. This visual feedback never moves the physical pointer or changes browser input delivery.

Keep browser chrome and native dialogs on the normal UIA/PX ladder in this file. This includes tabs, the address bar, menus, permission prompts, downloads, file pickers, and authentication windows. Avoid Ctrl+L, tab switching shortcuts, Start-Process, and shell activation paths when the user expects background operation.

WebView2 inside a non-browser host is not automatically equivalent to a standalone Edge target. Use typed browser mutation only when get_browser_state returns an exact binding and mutation_allowed:true for the selected (pid, window_id). Otherwise use the native UIA/PX route or accept the structured refusal. Firefox page mutation is not supported by the typed browser tools yet.

Common failure modes (Windows-specific)

• Session 0 daemon — qwen-cua-driver doctor reports SessionId: 0. UIA enumeration returns empty, screenshot returns blank. Fix: stop the daemon, kick the autostart task with schtasks /Run /TN qwen-cua-driver-serve.
• Stale HWND (Invalid window handle 0x80070578) — the window was closed, re-created (e.g. UWP shutdown-on-idle), or moved to a different desktop session. Re-resolve via list_windows.
• Calc display stuck at "0" after pixel clicks — the (x,y) UIA hit-test missed and PostMessage fell through (PostMessage is a silent no-op on UWP). Switch to element_index mode. Symptom: the action result reports route:"synthetic_events" instead of route:"accessibility".
• LibreOffice (VCL) type_text / hotkey reported success but nothing happened — VCL/SAL apps route accelerators through TranslateAccelerator (reads GetKeyState, which PostMessage doesn't update) and the Calc/Writer document grid only takes real keystrokes when a cell is in edit mode, so background WM_CHAR / key-combos are silently dropped. Two honesty mechanisms now cover this instead of a blind success:
  • hotkey / press_key (keystroke + key-combo): delivery_mode:"background" surfaces a background_unavailable error for VCL.
  • type_text does a UIA read-back and returns the shared ActionResult: effect:"confirmed" with evidence:[{"kind": "value_readback"}] when the value reflects the text, and effect:"unverifiable" when the value is unchanged or unreadable. Use the optional escalation to choose the next rung. Pass an element_index for reliable verification: the read-back then reads that specific element by handle (ValuePattern → TextPattern), which is focus-independent — it can confirm or disprove a change whether or not the target is foreground. (Verified live against the WPF harness: typed via element_index, read back confirmed, value independently present in the next snapshot — app never fronted.) Without an element_index it falls back to system-wide GetFocusedElement, which on Windows only resolves when the target is the foreground app (no per-app AXFocusedUIElement like macOS); a backgrounded target then reads an unverifiable result even when the text actually landed — so it is NOT a failure signal; call verify_state or inspect a fresh screenshot. Escalate to delivery_mode:"foreground" for both (SendInput Unicode / accelerator). But foreground needs the swap to actually land — if the daemon lacks UIAccess and bring_to_front returns landed_on_target:false (or it reverts before the next call), you can't drive it by input at all: produce the artifact and launch_app it (build the .xlsx / .docx and open it) rather than typing into the GUI.
• Edge / Chrome shows tab switching even though I used pid-scoped hotkey — Ctrl+Tab / Ctrl+1..9 aren't pid-scopable; the receiver activates. Use the windows-per-URL pattern.
• Get-StartApps returns no AUMID for an app I see in Start Menu — the app might be a Win32 desktop app, not a packaged app. Use {path: "..."} instead of {aumid}. (Win11 Calculator IS packaged; Win10 classic Notepad was not.)
• launch_app returns pid N but list_windows({pid: N}) returns empty — UWP cold-launch race: the AppFrame HWND hasn't materialized yet. Re-call list_windows({pid: N}) after 500ms; for chronic cases, key off the app name in list_windows({}) output.
• JPEG screenshot has more compression than expected — default quality on the MCP screenshot compat path is 85; for raw qwen-cua-driver call screenshot, defaults to PNG (no compression). Pass {format: "jpeg", quality: 70} to opt into compressed screenshots. The max_image_dimension config (default 2048) downscales via Lanczos3 before encoding.

Diagnostics

qwen-cua-driver doctor reports:

• Daemon version and install paths
• Current session ID (must be ≥1)
• COM apartment status (STA / MTA / uninitialized)
• UIA reachability (can we connect to CUIAutomation?)
• AppX broker reachability (for packaged-app activation)
• PATH state (is qwen-cua-driver actually on PATH?)
• Autostart Scheduled Task status

Run it whenever a tool call returns unexpectedly. Most failures trace back to one of these checks reading "false."

qwen-cua-driver autostart status reports whether the daemon is registered to auto-start at logon AND whether it's currently running:

• not-registered — Task Scheduler explicitly reported that the named task does not exist. Re-register via qwen-cua-driver autostart enable.
• registered (not running) — autostart task exists but no daemon process. Kick it with qwen-cua-driver autostart kick.
• registered (running) — happy path.
• permission-denied — the current process cannot inspect Task Scheduler; registration is unknown. Re-run the status check from a context that can read the task rather than re-registering it blindly.
• unknown — the query failed for another reason. The command exits non-zero and includes the original schtasks diagnostic; do not treat this as an absent registration.

Recording

Windows recording uses ffmpeg with gdigrab and writes the same trajectory shape described in RECORDING.md. Install ffmpeg on PATH for MP4 capture. When it is unavailable, per-turn state and screenshots continue and last_error reports the missing dependency.