docs/internal/web-ui.md
Purpose: the state of the non-terminal (web) frontend — the unified-scene architecture it is built on, what ships and works today, and a consolidated analysis of the design gaps (decisions still to be made) and implementation gaps (work items inside the current design) between the current prototype and the polish bar set by desktop-class web editors, VS Code in particular.
This doc replaces the retired UNIFIED_SCENE_DESIGN.md and NON_TERMINAL_UI_RESEARCH.md (in git history) as the referenced design home for the web UI. The evaluation evidence behind the "verified" claims is web-ui-rendering-eval.md at the repo root (2026-07: full build, the then-50-assertion Playwright suite, the scene_parity test, and custom headless-Chromium probes).
Status (2026-07-06). Since the evaluation, three waves landed on top of it: the test suite was packaged and wired into CI (§4 Testing & tooling); the Phase-A input/interop work shipped (desktop IME text sink, OS clipboard both ways, click-count forwarding — §3.5/§3.6); and the §3.1 transport was rewritten to a WebSocket pushing scene region diffs (input batching and the unchanged-scene short-circuit fell out of it). Release-build serving is now the documented default for interactive use, with measured release numbers in §3.1. A fourth wave landed the remaining Phase-A/B frontend items: per-region DOM patching (§3.4), measured font metrics + app zoom (§3.3), and touch pan/scroll on mobile, plus TUI-parity placement fixes (dropdown flush under the menu bar, palette as a bottom sheet) — the suite now stands at 72 assertions. The biggest open items, in rough order of value: per-row patching inside panes (§3.4), then the Phase-C reach work (preedit overlay, accessibility, auth/multi-session).
The architecture is described operationally in web-ui/README.md and, for the render-pipeline side, in rendering-and-layout.md. The short version:
Scene projections on Editor) and consumed by both the TUI cell renderer and the web bridge. A Rust parity test drives one editor through both and fails if they disagree on what the chrome is.Editor::render into an in-memory buffer with chrome-cell drawing suppressed (suppress_chrome_cells), slices pane interiors (line-number gutter as its own block), and the frontend draws them as SVG text with every glyph pinned to its exact cell column. Layout, wrapping, highlighting, folding, scrollbars and split geometry are all the pipeline's own.handle_key / handle_mouse / shared hit-dispatch paths at cell coordinates; the browser re-renders from the editor's new state. The frontend holds no editor model./ws) carries everything: the server's event loop ticks the editor (drains async LSP/plugin/file events, steps animations) at ~40 ms while the scene reports activity and ~250 ms when idle, and pushes a region-diff frame only when the scene actually changed (§3.1). All input rides the same socket; the HTTP routes stay alive for curl and the parity harness.Verified working end-to-end (all 57 bundled Playwright assertions — now exercising the WebSocket input path and the push/diff/single-client behavior of §3.1 — plus custom probes): every chrome surface renders as native HTML with zero cell/SVG leakage; splits, drag selection, wide CJK/emoji glyphs, theme switching (native chrome restyles from the live Theme, light and dark), and a genuinely capable mobile touch shell (sticky one-shot modifiers, soft-keyboard summoning, breakpoint re-fit).
Strengths to preserve through any of the work below:
color-mix), so even high-contrast terminal palettes read as a designed UI."Polish" is not one feature. Measured against VS Code in a browser (vscode.dev / code-server), the bar decomposes into:
Two things VS Code has that are editor-core feature gaps, not web-frontend gaps, and therefore out of scope here: the minimap and sticky scroll (neither exists in the TUI; the web renders what the pipeline renders). Pixel-smooth sub-cell scrolling is likewise bounded by the cell model itself — the web can only scroll in whole visual rows for as long as the pipeline thinks in cells. These belong to core rendering discussions, not this doc.
These are the places where reaching the bar requires choosing an architecture, not just writing more of the current kind of code. Each lists the tension and a recommended direction.
The recommended end state is now the shipped transport. The bridge (crates/fresh-editor/src/webui/mod.rs) runs one nonblocking event loop on the editor's thread (the editor is not Send; nothing moved off-thread):
GET /ws upgrades to a hand-rolled RFC 6455 WebSocket (matching the bridge's hand-rolled HTTP; sha1 + base64 for the handshake, 7/16/64-bit frame lengths, masked client frames, ping/pong, close echo, defensive continuation handling).{"type":"hello","seq":0,"scene":<full scene>} — the same object GET /state returns, clipboard field included. A reconnect (or server restart) gets a fresh hello; seq restarts.{"type":"frame","seq":N,"changed":{<path>:<value>,…}} where paths are "w", "h", "theme", "clipboard", and "regions.<key>" — except panes, which diff one level deeper ("regions.panes.<i>" plus "regions.panes.len" when the count changes), since panes carry the bulk of the bytes: typing resends only the changed pane. A changed value replaces the old one wholesale (null is legal — regions are frequently null). Change detection compares each unit's serialized JSON against the last-sent copy, cached per session; the cache is written only by the WS push path, so HTTP /state calls can't confuse it."type":"key"|"mouse"|"action"|"widget"|"settings"|"kbedit"|"paste"|"resize". Both transports run through one shared dispatch (apply_message), so they cannot drift. Inbound frames are drained as a batch and applied in order with one render per batch, not per message — the input batching §3.2 asked for fell out of the transport.poll.active hint is true, ~250 ms when idle, client or no client (async LSP/plugin/file events never stall). The scene build+diff is additionally gated on editor_tick's own needs-render signal, so a connected-but-idle session measures ~1.6 % CPU (debug build; ~0.7 % with no client).409 Conflict before any upgrade; an Origin whose host differs from the bind host gets 403 Forbidden (non-browser tools send no Origin and pass). Multi-session and real auth remain §3.7.Connection: close) — curl-ability and the parity harness (/step, /reset) are untouched. A mutation made over HTTP is pushed to the connected browser as a diff in the same loop pass.Measured (140×44, loopback; 20 samples per probe, same probes for both profiles, 2026-07-06). Debug build: key→frame RTT over WS min 55.5 / avg 62.9 / max 92.7 ms (an earlier run measured 51.8 / ~60 / 77.8) vs the same box's HTTP /key avg 64.1 / max 142 ms; GET /state min 53.0 / avg 60.2 / max 72.1 ms — the transport overhead over a bare scene build is a few ms, and the remaining latency is the debug-build render itself. Release build — now the recommended way to serve interactively (cargo run --release …; PROFILE=release web-ui/test/run.sh for the suite; §4): key→frame RTT over WS min 14.8 / avg 16.7 / max 18.8 ms; GET /state min 23.0 / avg 25.6 / max 26.3 ms — the server round-trip stops being the bottleneck. End-to-end in headless Chromium (key sent via window.fresh.sendKey, awaiting the pushed-frame counter — i.e. including the page's own paint): debug min 51.2 / avg 74.2 / max 113.1 ms; release min 17.9 / avg 42.1 / max 66.6 ms, so in release the tail above the ~17 ms socket RTT is the frontend's own render (§3.4's per-row patching is the remaining lever). Idle CPU over 10 s with a connected idle client: ~1.1 % release, ~1.5 % debug; zero frames pushed when nothing changes.
Still PLANNED from the original option list: gzip/deflate (less urgent now that idle traffic is zero and typing sends per-pane diffs), and the multi-client / auth posture of §3.7.
The founding rule — the frontend re-implements nothing — means every echo of a keystroke waits on the server round-trip. VS Code (local) is in-process; VS Code remote solves the same problem with speculative local echo (type-ahead rendered locally, reconciled when the server catches up), which is a deliberate, bounded re-implementation.
Decision to make: how far to push honest latency before considering speculation. Recommended order: (1) release-build serving IMPLEMENTED/measured (release is the documented default; §3.1 numbers — WS key→frame p95 ≈ 19 ms in release, browser-measured end-to-end p95 ≈ 66 ms with the excess being frontend paint), (2) WebSocket transport IMPLEMENTED (§3.1 — removes per-request connection setup), (3) input batching IMPLEMENTED (the WS loop drains all pending input as one batch → one editor pass → one render). Only if measured p95 echo still exceeds ~50 ms should speculative echo be designed — and then as a clearly-labeled, reconciled overlay (the cursor cell + inserted glyphs only), never as a second text model. Getting this wrong quietly forfeits the architecture's core guarantee. Verdict with release serving measured: the server round-trip — the only thing speculative echo could hide — is ~17 ms (p95 ≈ 19 ms), comfortably under the ~50 ms bar, so by this section's own criterion speculative local echo is NOT warranted; the residual end-to-end tail is the page's own paint, which speculation would still have to pay (§3.4 per-row patching is the honest fix).
The measured-metrics + zoom half of this section is now shipped; the grid is no longer hardcoded. At boot — and again on every zoom or devicePixelRatio change (matchMedia re-arm pattern, catching browser zoom and monitor moves) — the frontend measures the real glyph advance of the SVG's own monospace stack via canvas measureText and derives the cell size from it: CW = measured advance × the shipped cell/advance ratio (so the default stack renders pixel-identical to the old constants at zoom 1.0 — verified byte-identical screenshots — while a differently-resolved font re-derives the whole grid), CH keeps the shipped line-height:font-size ratio. The measured values feed every cell↔pixel mapping — placement, the SVG viewBox/font-size, caret, hit-testing (cellAt divides by the live values, so clicks stay exact under zoom) — and a resize re-fits cols/rows whenever they change. App zoom is Ctrl+= / Ctrl+- / Ctrl+0 and Ctrl+wheel, frontend-owned keys that never reach the editor (they change view typography only; the editor sees just the cols/rows re-fit); the factor is persisted in localStorage as a pure view preference, never editor state. window.fresh.metrics exposes {cw, ch, font, zoom} for the test suite.
Still PLANNED: a user font-family setting, and the ligature story — per-glyph x-pinning (chosen so runs can't drift) is fundamentally ligature-hostile; supporting them means pinning per run and accepting measured cluster widths, or explicitly documenting ligatures as out of scope for cell-grid fidelity. Recommended: per-run pinning behind a setting, ligatures deferred.
Today: one <svg> per pane, one <tspan> per style run with a per-character x list, vector rules replacing box-drawing glyphs. Visually excellent (verified crisp at 2× DPR, exact column alignment, no run-boundary drift) and fine at 140×44. It scales linearly with visible cells and rebuild frequency; a 4K fullscreen grid with several splits multiplies both.
Per-region DOM patching is IMPLEMENTED (the frontend half of the §4 performance item): #app holds one stable display:contents container per region family — panes one per index — and applyFrame maps each frame's changed paths to the set of dirty regions, rebuilding only those containers. A typing frame re-emits one pane's SVG and touches nothing else in the DOM (scroll positions, caret blink phase and IME focus survive by construction; the caret is additionally patched in place so unrelated frames don't restart its blink). window.fresh.renderedRegions exposes the rebuilt set for the suite. Per-ROW patching inside a pane's SVG is the remaining PLANNED step:
Options: keep SVG but patch per row (keyed rows, only re-emit changed ones — the bridge already sends rows as arrays, so row-level dirty detection is a string compare); absolutely-positioned DOM line divs (what VS Code uses — cheap partial updates, native selection possible, but sub-pixel column alignment is harder); canvas/WebGL (xterm.js route — fastest, but forfeits DOM text, accessibility and crisp zoom for free).
Recommended: SVG with per-row keyed patching as the next step (smallest delta, keeps every current fidelity property), canvas only if profiling of realistic large-viewport use demands it. Whatever the medium, the switch must not leak into the bridge contract — cells in, pixels out.
Buffer selection is entirely editor-side (drag → real selection cells, rendered as background runs — verified). Two further selection surfaces are IMPLEMENTED on top of that:
Live-terminal selection — IMPLEMENTED core-side (TUI + web). A live terminal grid has no cursor/selection model, so a left-button drag on it (recorded as MouseState::terminal_drag_pending on the down — a bare click still only focuses, preserving click-to-focus-and-type) drops that split into read-only scrollback, exactly the Ctrl+Space / scroll-up transition. sync_terminal_to_buffer pins the scrollback viewport to the first byte of the just-appended visible screen, so the view is pixel-identical to the grid and both the press origin and the drag position resolve to exact byte positions by line arithmetic (begin_terminal_grid_selection in app/terminal_mouse.rs); the standard text-selection drag machinery then takes over, Ctrl+C copies through the editor clipboard (→ browser via the scene clipboard field below), Ctrl+Space resumes the shell. Double-/triple-click word/line selection works in the scrollback view (still inert on the live grid). jump_to_end_on_output yields while a selection drag is pending/active or a selection exists, so chatty programs can't yank the view mid-copy. Mouse forwarding to the inner program is now subscription-based rather than alternate-screen-based: button/drag events forward only when the program enabled mouse reporting (DECSET 1000/1002/1003), buttonless motion only under all-motion (1003), wheels additionally under the old alternate-screen rule (alternate-scroll arrow synthesis for pagers is unchanged) — and Shift bypasses forwarding entirely (xterm convention), so text is always selectable even under a mouse-hungry program. Both are configurable: terminal.mouse_drag_selects = false keeps the live grid inert under drags, and terminal.mouse_forwarding = "alt_screen" restores the legacy forward-everything-to-alt-screen rule.
Alt-hold native browser selection — IMPLEMENTED (web frontend only). user-select:none plus preventDefault-everywhere normally forecloses browser-native selection. Holding Alt flips ownership: capture-phase guards stop every forwarding handler (element- and document-level, without preventDefault), body.natsel re-enables user-select (line-number gutters stay excluded — the gutter/text split exists for exactly this), and the browser's own drag/double-click selection works over ANY text — buffer SVG, live terminals (without pausing them), file explorer, menus, dialogs. Ctrl+C (or right-click → Copy; the context menu stays native while Alt is held) copies it; natselText() rebuilds SVG cell-grid selections row-aware, since grid rows are separate <text> elements that Selection.toString() would run together, and trims the grid's cell padding. Alt is free to own: the editor's menu accelerators are Alt+letter chords (still forwarded) and its mouse paths only read Shift/Ctrl. A selection left behind is cleared by the next ordinary mousedown. Known limitation: a pushed frame that rebuilds the pane under a still-held native drag collapses that DOM selection — the core-side terminal path above is the robust route for streaming terminals; Alt is the universal escape hatch for chrome and glance-copies. Ctrl+Alt+S toggles the whole mode off/on (a frontend view preference persisted in localStorage, same class as zoom / palette placement) for users whose workflow clashes with Alt-drag.
Inbound (paste) — IMPLEMENTED. A document paste listener reads clipboardData and delivers the whole text in one paste message over the WebSocket (the POST /paste route stays for curl/harness use); the bridge runs the editor's real bracketed-paste path (paste_bracketed_into_focused_panel → paste_text, the same routing the terminal's Ev::Paste takes), so focused panels, prompts, terminals and column-mode paste behave exactly as in the TUI. The bare Ctrl/Cmd+V keystroke is deliberately not forwarded (and not preventDefault-ed): the DOM paste event that follows is the one paste path, mirroring how the terminal frontend receives a bracketed paste rather than a Ctrl+V key. Long pastes never loop through per-char /key posts.
Outbound (copy/cut) — IMPLEMENTED via the scene-field approach (the editor stays the source of truth for what "copy" means, including block selections and multi-cursor): browser-facing scenes carry "clipboard": {"seq": N, "text": "..."}, where seq increments whenever the editor's internal clipboard text changed — the bridge tracks a hash per server (ClipboardSync in webui/mod.rs), so no new core-editor state; the core only gained the read-only Editor::clipboard_text() accessor. The frontend calls navigator.clipboard.writeText when it sees a new seq; the write happens while handling the pushed frame that answers the very Ctrl+C / menu click that ran the copy, i.e. inside that gesture's transient-activation window (a reconnect hello adopts the seq without writing, so a stale copy is never replayed). The exposed text is capped at 1 MiB and never logged; a rejected write (no permission, non-secure origin) degrades silently to editor-internal copy/paste. Still PLANNED: a dedicated fetch for over-cap copies, and rich-text (copy_html) pass-through.
Hidden-input text sink — IMPLEMENTED as the universal text path. The mobile shell's hidden input is now created and kept focused on desktop too (focus({preventScroll:true}); refocused on non-chrome clicks and window focus; visually hidden but never display:none, which would break IME). Desktop keeps keydown for plain printable keys, shortcuts and the named-key allowlist — that handler preventDefaults, which suppresses the matching beforeinput, so there is no double delivery and the mobile-only kdHandledAt de-dupe window never needs to engage on desktop. Text the keydown path can't express arrives through the sink: composition-less insertText via the existing beforeinput translation, and IME composition (CJK candidate commit, dead-key accents — key arrives as "Process"/"Dead") via compositionend, with isComposing guards on keydown/beforeinput/input so the IME owns the sink until commit.
Still PLANNED — the preedit story: the uncommitted composition currently lives invisibly in the hidden input; real IME polish means rendering it inline at the caret, which needs a small new scene affordance (a transient "preedit overlay" the frontend paints at the cursor cell — deliberately not an edit to the buffer). Also still open: browser-reserved shortcuts (Ctrl+W/T/N are uninterceptable in a normal tab — the honest fixes are the Keyboard Lock API (fullscreen/PWA only), an installable PWA shell, and a documented alternate-binding fallback). AltGr and autorepeat fall out of the implemented paths for free.
Today: plain HTTP, localhost, no auth, single thread, one implicit client, README warns accordingly. That is the correct scope for a dev prototype and the code says so honestly.
Decision to make only if the web UI graduates: session model first (one editor per client vs. shared editor with presence), then the boring hardening (token auth, TLS or bind-behind-reverse-proxy, reconnect/resync semantics beyond the implemented full-scene hello). The trap to avoid is accreting multi-client behavior onto a shared-mutable-editor accident. The §3.1 transport already takes the two defensive steps this section asked for while the decision is pending: concurrent WebSockets are actively rejected (409 Conflict — never interleaved), and upgrades from a foreign Origin host get 403 Forbidden. Everything else here stays PLANNED, and the single-client stance stays loud in the docs.
Today: effectively none. Buffer text lives in SVG <tspan>s (invisible to most screen readers as structured text), chrome divs carry no ARIA roles, and the stylesheet suppresses focus outlines globally.
The semantic-scene architecture is actually an unusual advantage here — the chrome is already a typed model (menus with enabled/checked state, rows, dialogs), so emitting role="menu"/"menuitem"/"dialog"/"tree" and aria-* from the projections is mechanical in a way it never is for pixel-first editors. The buffer needs the VS Code trick: a hidden accessible region mirroring the caret's line/viewport as plain text. Decision to make: whether the accessible-text mirror is a frontend derivation from cells (cheap, approximate) or a first-class scene projection (correct, one more surface under parity discipline). Recommended: scene projection — it's the same single-source principle, and the TUI could use it for its own screen-reader story later.
Work that is well-defined inside the current architecture. Effort: S (hours), M (days), L (week+).
Input & OS interop
paste event handler → editorpaste listener → one paste message over the WebSocket → the editor's bracketed-paste path (see §3.5; no per-char key loops, and the bare Ctrl+V key is not forwarded).beforeinput + compositionend), the preedit overlay of §3.6 remains PLANNED.count (event.detail); when count ≥ 2 the bridge primes the editor's own click-tracking state (previous_click_time/position, click_count) so detect_multi_click resolves the browser's count deterministically — the editor's word/line-selection path itself is untouched, and no timing/cell-slop mismatch across the HTTP hop can drop a double-click.scrollup/scrolldown/scrollleft/scrollright at the touch-start cell, one step per whole cell panned, plain 1:1 with no momentum; taps stay untouched so the browser's synthetic mouse events keep driving the unchanged tap-to-position-cursor click path, and preventDefault fires only mid-pan so chrome panels keep native touch scrolling and the page never rubber-bands.Rendering & visual polish
:focus{outline:none} with visible focus styling on interactive chrome (prerequisite for §3.8, independent of it). (S)Performance (stepping stones toward §3.2; §3.1's transport is done)
fresh --web …, or cargo run --release --features web -p fresh-editor --example webui_server -- …), and the suite runs against it with PROFILE=release web-ui/test/run.sh (debug stays the CI default). Measured release vs debug (140×44, loopback, §3.1): WS key→frame RTT avg 16.7 / max 18.8 ms vs avg 62.9 / max 92.7 ms; GET /state avg 25.6 ms vs avg 60.2 ms; idle CPU with a connected client ~1.1 % vs ~1.5 %. The server no longer dominates the echo — the remaining tail is frontend paint (§3.4 per-row patching).permessage-deflate) — less urgent post-§3.1, still worthwhile for big frames like theme switches. (S)applyFrame maps each frame's changed paths to dirty region containers and rebuilds only those (panes per index) — a typing frame re-emits one pane's SVG and leaves the rest of the DOM untouched (window.fresh.renderedRegions). Per-row patching inside panes (§3.4) is the remaining step. (M)/key batch → one render.Testing & tooling
web-ui/test/package.json declaring playwright (the documented test command currently fails from a clean checkout with ERR_MODULE_NOT_FOUND) plus a one-command runner that builds the bridge, starts it, runs drive.mjs, and tears down.web-ui/test/package.json pins playwright, and web-ui/test/run.sh builds the bridge, installs deps, starts/polls the server, runs the suite, and tears down (CI: .github/workflows/web-ui.yml).scene_parity on PRs touching web-ui/ or the view/webui crates..github/workflows/web-ui.yml runs both jobs (Rust cache + a ~/.cache/ms-playwright cache keyed on the pinned playwright version) on pushes/PRs touching web-ui/**, the webui bridge, scene.rs, the parity test, or the workflow itself.color-mix and per-glyph SVG need at least a Firefox and WebKit smoke pass. (S/M)Docs
UNIFIED_SCENE_DESIGN.md, NON_TERMINAL_UI_RESEARCH.md) in web-ui/README.md and source comments, and the wrong example path in drive.mjs's header, are fixed alongside this doc's introduction; keep them pointing here.Each phase keeps the invariant that makes this frontend worth having: the editor remains the single source of truth, and anything semantic added for the web lands as a scene projection under parity test, never as web-only logic.