docs/plans/VOICE_IO.md
Status: Shipping — phases 1, 2, 4, 7 (macOS) complete · 3 partial · 5, 6, 7 (Windows/Linux), 8 pending Touches: backend, Tauri shell, frontend, a new native shim crate Last reviewed: 2026-04-21
Phase 1 — Groundwork. Audio tab retired from the sidebar; its device / channel
config lives under Settings. Captures tab is live at /captures with no feature
flag.
Phase 2 — Local LLM backend. LLMBackend protocol alongside the existing
TTS/STT backends. qwen_llm_backend.py, services/llm.py, routes/llm.py, and
a shared model-download / cache pipeline. Qwen3 0.6B / 1.7B / 4B registered and
user-selectable via capture_settings.llm_model.
Phase 4 — Captures tab. List + detail view, source badges (dictation /
recording / file), retranscribe, refine (flags + model resolved from a
server-side capture_settings singleton), delete, and the Play-as-voice
dropdown over every profile.
Phase 3 — In-app voice input. CapturesTab dictates end-to-end via
useCaptureRecordingSession, which the Phase 7 floating pill also consumes.
Outstanding: a universal mic button on other text inputs (Generate form,
profile descriptions, story titles, etc.), and the streaming
/transcribe/stream WebSocket — today's flow is a single POST /captures
with the complete audio blob.
Phase 7 — External dictation shell (macOS). Both halves shipped on macOS.
Hotkey half:
tauri/src-tauri/src/chord_engine.rs — pure state machine. Unit tests green.tauri/src-tauri/src/hotkey_monitor.rs — rdev-based global listener on a
background thread, with set_is_main_thread(false) applied to sidestep the
macOS 14+ TSM crash (Narsil/rdev#165).
Right-hand-only defaults preserve left-hand Cmd+Option+I devtools.Cmd+Option (push-to-talk) and
Cmd+Option+Space (toggle-to-talk). The PTT → Toggle upgrade transition is
preserved — adding Space mid-hold promotes the session without interrupting
audio.DictateWindow — transparent, always-on-top, borderless 420×64 webview
pre-created hidden at app setup. Shows on chord-start, hides on
capture-cycle completion. Error state on the pill auto-dismisses and
copies-to-clipboard on click.Paste half (macOS):
clipboard.rs — NSPasteboard snapshot that walks pasteboardItems and
copies every (uti, bytes) pair so multi-type content (images, styled
text, file refs) survives the round-trip. save_clipboard,
write_text, restore_clipboard, current_change_count.synthetic_keys.rs — CGEventPost at the HID tap with the full four-event
Cmd+V sequence (Cmd down → V down w/ flag → V up w/ flag → Cmd up).focus_capture.rs — AXUIElementCreateSystemWide +
AXUIElementCopyAttributeValue(kAXFocusedUIElement) +
AXUIElementGetPid, with the AX attribute key CFStrings built at
runtime because they're CFSTR macros, not linkable symbols.
NSRunningApplication.activateWithOptions: for re-activation.accessibility.rs — AXIsProcessTrusted gate.paste_final_text command — activate → 120 ms settle → save clip →
write text → ⌘V → 400 ms → restore. Skips when focus was in Voicebox
itself.dictate:start event payload; DictateWindow holds the
snapshot in a ref and consume-once-nulls on paste so a late-arriving
refine from an earlier session can't misfire.Outstanding: Windows SendInput / UIAutomation / SetForegroundWindow
equivalents, Linux uinput / AT-SPI equivalents (and the Wayland story),
first-run Accessibility prompt UI with deep-link to System Settings,
direct-injection path for focus-was-inside-Voicebox (step 6 — dictating
into our own Generate tab currently falls back to the capture list).
/speak endpoint, no
voicebox.speak MCP tool, no per-agent voice binding, no persona metadata
on profiles.mlx_backend.py).
Parakeet v3, Qwen3-ASR, Kyutai — all unregistered.These fell out of the Phase 3/4/7 work but deserve their own mention:
capture_settings and
generation_settings tables. The client sends nothing but the audio; STT
model, refine flags, refine LLM, and the auto-refine flag are all resolved
server-side, so sibling Tauri webviews can't go stale.POST /captures transcodes anything
librosa can decode (webm/opus, m4a, etc.) to WAV before handing it to
whisper, side-stepping miniaudio's format gaps inside mlx-audio.Called out in recent sessions but not yet in a phase:
push_to_talk_chord / toggle_to_talk_chord in capture_settings,
surface a chord-picker UI in CapturesPage, and wire a Tauri
update_chord_bindings command so HotkeyMonitor::update_bindings picks
up user changes live.Voicebox ships the output half of a voice I/O loop: clone a voice, generate speech, apply effects, compose multi-voice projects. The input half — speech to text, dictation, routing — exists today as a single Whisper model wired into the Recording & Transcription panel. This doc proposes making voice input a first-class pillar: more STT engines, a dictation shell (global hotkey, audio capture, paste, streaming), a local LLM backend, and a user-configurable pipeline from captured audio to whatever the user wants to do with it.
Positioning is the key move. Voicebox becomes the local voice I/O layer for humans and AI agents — a local alternative to cloud dictation tools, with the differentiator that we also do TTS and voice cloning. The same app that captures your voice can generate a response in any voice profile you've cloned. "Anything voice is Voicebox."
Before this plan, Voicebox was "the open-source AI voice cloning studio." Cloning was the headline capability.
After this plan, Voicebox is "the open-source AI voice studio." Cloning is one capability in a broader category that now spans input (STT, dictation), intelligence (local LLM, refinement, persona), output (TTS, cloning, effects, Stories), and routing. The word "cloning" drops out of the top-line descriptor because it's become a feature rather than the thesis.
Voicebox ends up covering the territory of two separately-funded, separately branded cloud incumbents that operate on opposite sides of the same voice I/O loop:
Both are cloud-only. Voicebox becomes the only local alternative to either, running in one app, with a single model directory and LLM shared between input and output. That bridging — dictation → LLM → TTS with a cloned voice in the middle — is the thing no single incumbent can match, because neither has the other half.
These are not engineering tasks but should ride the Phase 4 ship so marketing and positioning stay in sync with the product.
STTBackend protocol already exists. The multi-engine registry pattern
shipped with TTS makes adding Parakeet v3 and Qwen3-ASR a days-not-weeks
effort on the backend side.1. Expanded STT registry. The existing STTBackend protocol abstracts
Whisper today. Add:
nemo_toolkit or transformers.transformers.All register via ModelConfig and use the same download, cache, and model
management UI we already have for TTS. Zero special-casing.
2. LLMBackend protocol. Mirror of TTSBackend / STTBackend. First
implementations are Qwen3 0.6B / 1.7B / 4B running on the same PyTorch + MLX
infrastructure we already run. One runtime, one model cache, one GPU-memory
story.
Why not llama.cpp or ollama: we already have the dependency surface and the
model download UX. A second runtime fragments cache directories and model-status
UI. If CPU-only Windows latency becomes a problem we can revisit.
3. Streaming transcribe transport. Add /transcribe/stream as a WebSocket
endpoint alongside the existing HTTP /transcribe. Audio frames flow in,
partial transcripts stream back. Same FastAPI process, same loaded models. This
keeps dictation latency off the per-request JSON-encode critical path and lets
us ship real-time partial transcripts later without a protocol change.
Every captured audio event flows through the same shape: Source → Transforms → Sink(s). Users configure presets that bind a source to a transform chain to one or more sinks.
Source Transform Sink
────────────────── ───────────────── ─────────────────
Hold to speak ──┐ STT model Clipboard + paste
Tap to toggle │ Refinement LLM Capture history
Long-form recorder ├──▶ Persona LLM ──▶ File on disk
File drop │ Translation (later) HTTP webhook
API call (WS / HTTP) ──┘ MCP server sink
TTS loopback (persona)
Platform sinks (later)
Source → Transform → Sink is internal, dataflow-style vocabulary (same shape
as Unix pipes, Apache Beam, Kafka) — not user-facing. The UI surface will use
Voicebox-native language (see open questions).
Concrete preset examples this shape enables:
Every user-facing feature collapses into (source + transform chain + sinks). Meeting-style capture isn't a separate product; it's a preset. Competing tools hardcode integrations (Trello, Granola); we make routing user-configurable.
The parts Tauri doesn't handle cleanly, gathered in one Rust crate with a platform-agnostic API:
global-shortcut
plugin requires full combos. We need "hold right-cmd" or "hold ctrl" as
primitives. On macOS this means a CGEventTap on a background thread with
polling fallback for dropped modifier events; on Windows a low-level keyboard
hook; on Linux X11 + libinput, with Wayland as a known gap.AXUIElement on macOS, UIAutomation on Windows,
AT-SPI on Linux. Check the element's role to decide between a direct
injection, a clipboard + paste, and a clipboard-only fallback with a
notification. A blind paste that only "works when a text field happens to
be focused" is the easy default; we should make the decision deliberately.Main process owns this crate. Webview never sees platform differences.
The paste sink adapts to what's in focus. This is a single sink type with branching behavior, not four separate sinks.
| Target | Delivery strategy |
|---|---|
| Focused text field inside Voicebox | Direct React state update via event. No clipboard involved. |
| Focused text field in another app | Accessibility-verified paste: save clipboard, write transcript, simulate paste, restore clipboard. |
| No text focus detected | Clipboard only, toast notification ("Transcript copied — no text field focused"). |
| Platform-specific special cases (terminal apps, specific editors) | Per-app overrides where the generic path misbehaves. |
| Concern | Layer |
|---|---|
| STT / LLM / TTS inference | Python backend |
| Model downloads, progress, cache | Python backend |
| Pipeline runner (orchestrates transforms and sinks) | Python backend |
| Audio capture from mic / system audio | Rust (Tauri side) |
| Audio streaming over WebSocket to backend | Rust |
| Global hotkey capture | Rust (native shim crate) |
| Paste simulation, clipboard save/restore | Rust (native shim crate) |
| Pipeline preset UI, capture history, settings | React |
Model work in Python. OS work in Rust. User config in React.
The current sidebar is Generate · Stories · Voices · Effects · Audio · Models · Settings. The existing Audio tab is output-device and channel routing
config — infrastructure, not a creative workspace — and the Settings page
already has a sub-tab pattern (ServerSettings/: Connection, Models, GPU,
Update) that fits it naturally.
Move Audio to a Settings sub-tab. Reclaim the sidebar slot for voice input.
The new tab shows recent captures (audio + transcript paired), active presets, dictation settings, model pickers for STT and LLM. Exact name is an open question.
Sidebar placement: Captures sits at position 3, directly under Stories and above Voices. Creates an "input voice / output voice" adjacency — captured speech is one slot away from the voices you can play it back through, which mirrors the Phase 4 "Play as voice" feature's mental model. Full order: Generate · Stories · Captures · Voices · Effects · Models · Settings.
The Captures settings page gets a "What's different" aside that introduces Voicebox's dictation story. The Generate tab deserves a parallel — first-time users need to be told what voice generation is for in a post-Voice-I/O world, not just handed a text field.
Shape: an empty-state card rendered in the Generate tab when there's no generation history yet, disappearing once the user has generated anything. Teaches without claiming permanent real estate. Parallel bullets to the Captures aside so the two tabs feel like two sides of one product:
This lands in Phase 4 alongside the Captures tab, for visual and thematic symmetry. Not a persistent sidebar — the Generate tab is a workspace and should reclaim its space once the user is producing work.
Every capture saves the original audio alongside the final transcript in a
pattern that mirrors data/generations/. Optional retention setting. Free for
us — the storage and UI patterns exist today for generations.
The WebSocket transcribe endpoint is a first-class public API, documented
alongside /generate. Pipeline presets are addressable by ID via
/pipelines/{id}/run so agent harnesses and shell scripts can invoke
user-configured flows. An MCP server sink ships built-in, so integrations with
Claude Code, Cursor, Cline, etc. are one checkbox rather than a custom build.
Dictation is one half of the loop — user speaks, agent listens. The other half — agent speaks, user hears — is equally load-bearing and deserves a first-class primitive rather than being buried as a TTS loopback sink or a consumer read-aloud button.
The shape is a single new capability: any agent can call Voicebox to speak arbitrary text in a user-configured voice. The same pill that surfaces during dictation surfaces during agent speech, so the user always sees what's coming out of their machine.
MCP tool: voicebox.speak({ text, profile?, style? })
REST: POST /speak { text, profile_id?, style? }
Both accept an optional voice profile (defaults to the user's configured
default), an optional delivery-style string for engines that support it, play
audio through system output, and surface the pill in a speaking state.
Key design points:
recording / transcribing / refining / rest to include speaking — voice profile name, waveform in
the profile's color, visible duration. Same floating surface for both
directions so users have one mental model.speak() surfaces the pill. No headless "TTS daemon" mode.This primitive is what makes "Voicebox as voice layer for every agent on your
machine" a concrete shipping capability rather than marketing language. MCP,
ACP, and A2A integrations all slot into it — none of those agent protocols
need to know anything about TTS models, GPU placement, or voice profiles.
They call speak().
Relationship to the persona loop. The persona loop below is one use of
speak() — STT → LLM → speak(llm_reply). Other uses skip STT entirely: a
long-running task announcing completion, a notification, an agent proactively
asking the user a question. The primitive is deliberately simpler than the
persona loop so it can serve both flows from the same API.
A capture and a voice profile sample both hold audio + text, so there's an
obvious temptation to unify them. Don't. The metadata and lifecycle
differences are real:
| Capture | Voice profile sample | |
|---|---|---|
| Profile association | Standalone | Bound to one profile |
| Text field | Raw transcript + optional LLM-refined version | Exact reference_text only |
| LLM refinement | Often applied | Must not be applied — the reference text must match the audio verbatim or cloning breaks |
| Volume | Dozens per day | ~5 per profile, semi-permanent |
| Typical content | Whatever the user said | Often scripted phrases for cloning |
A unified table would mean nullable profile_id, nullable refined_transcript,
nullable reference_text — a fat row that means different things in different
states. Not worth the complexity.
What to ship instead: a one-way promote action. Capture → Sample, zero data-model churn. Thin endpoint:
POST /profiles/{id}/samples/from-capture/{capture_id}
Reads the capture's audio path and raw transcript, calls the existing
add_sample() service with reference_text pre-filled from the transcript,
lets the user edit the reference text in a dialog before saving (transcripts
are usually 90% right but cloning wants 100%). The capture stays in the
Captures tab untouched — the sample is a copy, not a move.
UI hook: the Captures tab's Send-to menu gains a "Use as voice sample…" option that opens a profile picker (with "+ New voice" for cold starts) and a reference-text confirm dialog.
The inverse direction (sample → capture) we deliberately skip. Samples are often scripted phrases used for cloning and they'd clutter the Captures list without adding value; also a subtle privacy surprise for users who don't expect their sample text browsable alongside real captures.
Audio storage deduplication is a later optimization. Today a promoted
capture duplicates the audio file on disk. That's fine. Content-addressable
storage (data/audio/<sha256>.wav with refcounting) can come in Phase 8 as
housekeeping — it'd let a capture and a sample share one underlying file, but
it's not user-visible and not necessary to ship the promote flow.
One flow on top of the speak() primitive: STT → persona LLM →
speak(llm_reply). Voice profiles gain optional metadata — a natural-language
personality description and default LLM behavior. The LLM runs text through
the profile's voice context, then speak() generates TTS with the cloned
profile. End-to-end voice-to-voice with a cloned identity transforming the
content, not just reading it.
Use cases this unlocks:
The shape — STT + LLM + TTS — also stages us for end-to-end speech LLMs which collapse all three into one transform. See Voice-to-voice readiness below.
The STT → LLM → TTS chain that powers the persona loop is a staged approximation
of voice-to-voice. A real end-to-end speech LLM (Moshi, GLM-4-Voice, Qwen2.5
Omni, Mini-Omni, Sesame CSM) replaces the three middle boxes with a single
fused transform: audio in, audio out, no text in between. The pipeline shape
accommodates this natively — register the model as a single LLMBackend (or
a new SpeechLLMBackend if the protocol needs to differ), expose it as a
transform type, and the same sinks work unchanged.
Framing this plan as "voice-to-voice scaffolding, with today's models as the staged fallback" is a strong pitch for agent-harness users who are already tracking these models.
instruct field on TTS generation. "Flow" is
Zapier-coded. "Route" is too networking. Needs its own pass.The v1 prototype deliberately skips the hardest parts of the long-term plan (native OS shim, global hotkeys, paste injection, new STT models). Everything in Phase 1–4 is in-process code using Whisper (which we already ship) and the existing model infra. No CGEvent taps, no SendInput, no clipboard timing. The usual OS-level sprawl of a dictation stack is exactly what we sidestep by starting in-app.
ServerSettings/ gains one
more section). Audio is device/channel config, not a creative workspace.main and
iterate without shipping half-built UI to users.LLMBackend protocol alongside TTSBackend / STTBackend. Register Qwen3
0.6B / 1.7B / 4B via ModelConfig. Reuses the HF download path, cache
directory, and model management UI. MLX (4-bit community quants) on Apple
Silicon, PyTorch (transformers AutoModelForCausalLM) elsewhere, same as our
TTS split.
No new runtime. No llama.cpp, no ollama, no fragmented model cache.
A universal mic button on every Voicebox text input. Hold, speak, release — text lands in the focused field via direct React state update. No OS APIs involved; Voicebox owns the input.
Marquee use cases:
Backend: add /transcribe/stream WebSocket endpoint. Audio frames in, partial
transcripts out. Reuses the existing Whisper model in memory. Optionally routes
through the LLM from Phase 2 for light refinement.
Graduates the tab out from behind the feature flag. Shows recent captures (audio + transcript pairs), lets the user replay, re-transcribe with a different model, edit the transcript, and send the output through the LLM. Archival is automatic — every capture saves audio alongside transcript.
Includes the "Play as voice profile" action. This is the simplest version
of the persona loop and it lands here for free — no LLM involved, no new
backend endpoints, just a Captures-tab button that sends the transcript text
to the existing /generate endpoint with a user-selected voice profile and
plays the result. Category-defining differentiator from the v1 prototype
onward: Superwhisper and WisprFlow cannot do this because they have no TTS. Voicebox can, with one day of frontend wiring.
Keep it aggressively minimal on day one. A capture list, a detail view, a model picker, a Play-as-voice dropdown. Refinement prompt editing, correction dictionaries, per-source overrides — none of that ships here. They become Tier-2 work when someone actually asks for them.
Two features that together make "Voicebox as the voice layer for every agent on your machine" a shipping reality:
speak() primitive. New POST /speak endpoint and voicebox.speak
MCP tool. Any agent calls Voicebox to speak arbitrary text in a
user-configured voice; the pill surfaces in a speaking state. Settings
UI for default voice, per-agent voice binding (Claude Code → Morgan,
Cursor → Scarlett), and a global mute.speak() with an LLM step — STT → persona LLM
→ speak(llm_reply). Voice profiles gain optional personality metadata
and default LLM behavior. End-to-end voice-to-voice with a cloned
identity transforming the content, not just reading it.Phase 4 demoed the user-initiated direction of the loop (Play as voice). This
phase ships the agent-initiated direction, which is the category-defining
capability and the pitch that lands with agent-harness users. The persona
loop is one flow on top of the speak() primitive — notifications, proactive
agent questions, and task-completion announcements all use speak() directly
without the LLM in the middle.
Launchable headline moment for the "local voice I/O" positioning.
Parakeet v3 and Qwen3-ASR register as additional STTBackend implementations.
Optional: Kyutai ASR. Multilingual coverage upgrades (50+ languages). Whisper
stays as the sensible default.
Deferred to here because Whisper is already good enough for v1 and the model picker UI exists. Adding rows to it doesn't change the product shape.
Native shim crate (global hotkey with modifier-only support, focus introspection via OS accessibility APIs, paste simulation, atomic clipboard save/restore). Tauri-side audio capture streams to the same WebSocket endpoint Phase 3 already ships. Paste sink with target-aware delivery.
This is the feel-good phase. It's also the riskiest: paste timing, hotkey reliability, and cross-platform focus detection are all engineering problems that have to be nailed or the product doesn't work. Phase 3's success derisks the backend plumbing before we start it.
Multiple source types, user-configurable transform chains, multiple sinks per
preset. MCP server sink (the agent-harness play). HTTP webhook sink. File
sink. Developer-facing /pipelines/{id}/run endpoint. Preset editor UI in
the Captures tab.
Dual-stream recorder (mic + system audio) as a source type. Chunked STT transform with overlap-based deduplication. Summary LLM transform. Long-form capture becomes a preset, not a new tab.
Platform-specific sinks (Apple Notes on macOS, Obsidian, etc.) as opt-in integrations behind the generic sink interface.
Two pieces of existing docs/PROJECT_STATUS.md work become load-bearing here:
ModelConfig (bottleneck #6 in PROJECT_STATUS).
Parakeet's Core ML path is Apple-only; the PyTorch path is Windows/Linux.
Same gating mechanism that currently blocks shipping VoxCPM.Neither needs to complete before Phase 1, but both should complete before Phase 4 when user-configurable pipelines surface the differences to end users.