.agents/skills/desktop-ui-verify/SKILL.md
Use this skill from the ani repo root when a change needs desktop runtime evidence. Anything Android — including PC-style wide-screen layout checks, which run fine on an Android emulator — belongs to .agents/skills/android-ui-verify; only desktop-only code paths (Compose Desktop behavior, packaging, JCEF, VLC/native libraries, window chrome) need this skill.
The toolbox script is:
DESK=.agents/skills/desktop-ui-verify/scripts/desk.sh
The core loop mirrors the Android skill: act → screenshot → Read the PNG → decide next action. Never claim desktop behavior works without a captured window screenshot.
Two launch modes:
./gradlew :app:desktop:run starts the app from Gradle with app/desktop/test-sandbox as working directory. Quick, but misses packaging problems..app — catches runtime-image/JCEF/native-library packaging problems that run misses../gradlew :app:shared:compileKotlinDesktop # cheap compile-only gate first
$DESK build # createDistributable with a JCEF-capable JBR
build requires a JBR containing JCEF (jmods/jcef.jmod + lib/libjcef.dylib); find_jcef_jbr.sh auto-selects one and exports it as ANI_COMPOSE_JAVA_HOME. A full first build takes many minutes — run it in the background.-Pani.api.server=http://localhost:4394; for real API data start the repo-local server first (see ../ani-api-server skill animeko-server-local-test-server) — desktop reaches it as plain localhost.tail — the pipe replaces the build's exit code with tail's, so failures read as success. Use -q (quiet) and read the full output file, or check the artifact ($DESK app, mtime) before trusting a "successful" build.Contents/Frameworks (CEF + jcef helpers) copied next to the packed runtime's Home; app/desktop/build.gradle.kts does this for both createDistributable and createReleaseDistributable (fixed 2026-07-05 — before that, non-release distributables crashed ~4 s after launch with a SIGSEGV in libjcef.dylib FindClass).$DESK launch # opens the newest built .app, waits, prints the process
$DESK screenshot # activates + resizes window to 1440x900, prints PNG path — ALWAYS Read it
$DESK click 300 200 # window-relative POINTS — on retina, screenshot PNG pixel / 2
$DESK type hello # ASCII keystrokes into the frontmost window
$DESK key return # also: tab, esc, space, delete, arrows
$DESK quit
agent-attach puts a tiny -javaagent (source: scripts/input-agent/InputAgent.java) into the .app's jpackage cfg; it injects synthetic AWT events in-process via a loopback socket — the system cursor never moves, the app need not be frontmost, and no Accessibility permission is involved. Validated end-to-end on the real Ani Compose window 2026-07-10 (clicks closed a dialog, switched nav, opened search; type inserted text into the search TextField — all while another app was frontmost). Injected events land on the Skia canvas (inject replies -> org.jetbrains.skiko.SkiaLayer$1) and Compose routes them internally. Ani draws its own window chrome, so info reports content == frame:
inject coordinates = window points = screenshot PNG px / 2, no title-bar offset.
$DESK agent-attach # once per built .app, BEFORE launch
$DESK launch
$DESK inject info # window bounds + content-area origin/size
$DESK inject click 300 200 # window-CONTENT points = window points minus title bar (~28pt);
# screenshot PNG pixel / 2, then subtract (content.y - frame.y)
$DESK inject type hello # goes to focus owner, else last injected-click target
$DESK inject key 10 # AWT keycodes: 10=Enter, 27=Esc, 9=Tab, 37-40=arrows
inject type goes to Compose's internal focus, which follows injected clicks — click INTO the
text field
first (opening a screen is not enough: after opening search, typing did nothing until the field itself was clicked).
For window-id screenshots WITHOUT activating/normalizing the window (keeps the app in the background), capture directly: swift scripts/find_window_id.swift --pid $(pgrep -xn Ani) then screencapture -x -l <id> out.png. These PNGs include a shadow border (e.g. 3016x1936 for a 1440x900 window — subtract (PNG_w−2880)/2 = 68px per side before converting px/2 → points).
Fall back to click/type/cliclick below only if injection misbehaves. Injected hover (inject move) drives Compose hover states from event coordinates, but anything reading the REAL cursor position will disagree.
Screenshots capture the window by CGWindowID (via find_window_id.swift), so they show the app's own content even when other windows overlap it — the user may be actively using the desktop. The AppleScript step first normalizes the window to position (40,80) size 1440x900 points.
Coordinate mapping: retina PNGs are 2x, so window point = PNG pixel / 2 (1440x900 window → 2880x1800 PNG; sanity-check your math against that).
Click immediately after a fresh screenshot: toasts and update dialogs auto-dismiss, and a click computed from a stale screenshot lands on whatever is underneath. click prints the AX element it hit (e.g. scroll area 1 of group 1 …) — read it to confirm the intended target.
click/type/key and the window-resize step drive macOS System Events and require Accessibility permission; screencapture needs Screen Recording permission (both verified granted for this terminal). On failure report the permission gap instead of concluding app breakage.
AppleScript click at can silently no-op on screens whose Compose accessibility tree is shallow (observed 2026-07-06 on the subject detail page: explore-page card clicks worked, then every detail-page click — buttons, back arrow — did nothing, with the AX hit always reporting the same generic UI element 1 of group 1 …). Fall back to real CGEvents: cliclick c:<screenX>,<screenY> (Homebrew-installed; screen = window point + window origin (40,80)) after set frontmost to true — clicked the same button first try.
cliclick is a GLOBAL click — it hits whatever is topmost at that point. If the user switches to another app mid-verification, every cliclick lands in their app (observed 2026-07-06: several clicks went into the user's Chrome session before a full-desktop capture revealed Ani wasn't frontmost). Immediately before each cliclick: set frontmost to true, then VERIFY with osascript -e 'tell application "System Events" to get name of first process whose frontmost is true' == Ani; if not, stop clicking and report. Also keep the window fully on-screen first (this display is 1728pt wide; a 1440-wide window moved to x=600 puts its right side off-screen where clicks silently miss).
The screenshot can capture the WRONG window. The process may own several CGWindows (e.g. a stale 1004x688 popup-host window stacked over the real 1440x900 main window, both at (40,80)); the frontmost-match then screenshots the static one, so the UI looks frozen and clicks look like no-ops even though they worked (observed 2026-07-06: playback had actually started while screenshots kept showing the explore page). Sanity-check every capture: PNG pixel size must equal AX window size × 2 (+136px shadow for screencapture -l); on mismatch, enumerate CGWindows by owner Animeko (swift CGWindowListCopyWindowInfo) and capture the right id via screencapture -l<id>.
There is no semantics/hierarchy dump on desktop (unlike Android's droid.sh tree); element targeting is visual. Prefer stable landmarks (window corners, sidebar order) and re-screenshot after every action.
Wait 1–3 s after actions; JCEF/browser content and network data need longer.
$DESK logs 150 # newest app log: ~/Library/Application Support/*Ani*/logs/, or app/desktop/test-sandbox/logs/ for dev runs
The app logs dataDir/logsDir on startup and installs an uncaught-exception handler that logs !!!ANI FATAL EXCEPTION!!! — grep for that when the app dies.
For bugs visible under a second (a flash of wrong state, a one-frame layout jump, loading flicker), screenshots are not enough — record the window and diff the frames:
REC=$($DESK record-start) # starts recording in background, prints the output .mov path
$DESK click ... / key ... # do the interaction that should (or should not) glitch
$DESK record-stop # stops (SIGINT-finalizes), prints the video path
$DESK frames "$REC" # scans all frames, prints change events + exported images
frames (= scripts/frame_diff.py, a symlink to the android skill's copy; needs ffmpeg on PATH) scores every consecutive-frame difference, groups spikes into events, and per event exports a contact sheet (frames tiled left→right, top→bottom — Read it to see the whole sequence at a glance) plus full-res frames named by video timestamp. How to work with it:
record-start returns — the window-id lookup takes a few seconds), so correlate by order/spacing, not wall clock. Mouse hovers create real events too (hover highlights, carousel arrows) — keep the pointer still when it shouldn't participate.$DESK frames <video> --around <t> --window 0.5; print every frame's score with --list.--crop 2880:1800:68:68 trims to exactly the screenshot framing; for a sub-region use --crop W:H:(x+68):(y+68) with screenshot-pixel values.--threshold (default 0.01, try 0.003).record <seconds> is the blocking variant for no-interaction captures (launch animations, video playback checks).$DESK quit unless the user wants the app kept open../gradlew :app:desktop:test
./gradlew :app:shared:compileKotlinDesktop
Other desktop Gradle tasks observed here: :app:desktop:runDistributable, :app:desktop:packageDistributionForCurrentOS, and release variants. For Compose screenshot assertions in tests, see utils/ui-testing: SemanticsNodeInteraction.assertScreenshot(expectedResource) is implemented for Skiko-backed (desktop) targets.
me.him188.ani.app.desktop.AniDesktop; System Events process name Ani (override with ANI_DESKTOP_PROCESS), but the CGWindow owner name is Animeko — window lookups therefore match by PID, not name.AppFolderResolver with app name Ani (Ani-debug in debug builds), observed at ~/Library/Application Support/me.Him188.Ani/.jcef Helper (GPU) etc.) next to the main process — pgrep -fl "Ani.app/Contents/MacOS" is a quick liveness check.app/desktop/build.gradle.kts supports ANI_COMPOSE_JAVA_HOME; the JBR must contain jmods/jcef.jmod and lib/libjcef.dylib./Users/him188/Library/Java/JavaVirtualMachines/jbrsdk_jcef-21.0.4/Contents/Home was verified end-to-end on 2026-07-05 (packages AND runs).mapfile, no declare -A).CommonKoinModule.kt ApiInvoker type mismatch) was verified FIXED on 2026-07-05: :app:shared:application:compileKotlinDesktop passes.