# 49 — Live voice: full-duplex spoken conversation Goal: replace the one-shot text-to-speech narration shipped in docs/design/38-voice-personality.md with a real spoken conversation — continuous mic, streaming transcription, spoken replies, barge-in — over a provider abstraction that treats Gemini Live, OpenAI Realtime, and a fully local stack (whisper.cpp + Piper/Kokoro) as peers, across desktop, web, and Telegram. Status: shipped foundation, redesigned 2026-09-23; realtime streaming and a physical-microphone acceptance run remain open. The **Shipped contract** section directly below is authoritative. The Architecture, Transport and Phasing sections after it are the original proposal, kept for its reasoning; where they disagree with the shipped contract (streaming `Listener` traits, a `VoiceRegistry`, a silence-emitting `LocalSpeaker`, `{"t":"barge"}` frames, a `[voice.local]` table, a shared `model` key), they describe something that was never built or has been removed. ## Shipped contract **One provider vocabulary.** `vak_voice::VoiceProvider` (`crates/vak-voice/src/provider.rs`) is the closed set `gemini`, `openai`, `local`, parsed in exactly one place. Every route resolves through it, so the socket, channel voice notes, `/voice/speak`, `/voice/providers`, `/config` and doctor cannot disagree about names. An unset provider is a configuration gap reported as such, never an implicit vendor. Hosted routes require explicit `transcription_model` / `synthesis_model` pins from the operator's discovered catalogue (invariant 9); there is no shared `model` fallback and no realtime model, because no realtime adapter exists. **One server surface.** `crates/vak-server/src/voice.rs` holds every voice route. Transcription has a single implementation used by the socket and by channel voice notes alike. Every paid call — a channel note, synthesis, each socket utterance — draws on one per-process `RequestWindow` sized by `voice.max_requests_per_minute`. `/voice/speak` records a `WorkReceipt` with real latency for every provider, success or failure. **Voice tiers.** The route a chat speaks and listens through is the workspace `[voice]` settings narrowed by its bot → chat `VoiceConfig` (provider and model pins win where set; a blank pin inherits), the same override-or-inherit chain as every other gateway tier (invariant 23). `/voice/speak` identifies the chat from the `session_id` it is given — the session's gateway binding — so a channel reply uses that chat's route, voice and persona without the bridge naming a chat key; the admin console's Preview passes its auditioned `voice_override` as the narrowest tier. A bot or chat tier is checked when it is written: an unknown provider name is refused then, not discovered when a voice note fails. **Channel voice notes.** Telegram, Discord and Slack bridges only attach the audio to `/gateway/inbound`. The gateway transcribes it after allowlist admission and request de-duplication, through the chat's tiers, so an unknown or pending chat never spends a provider call and a retried request is not transcribed twice. The transcript joins the typed text as the user's message; a note that could not be transcribed reaches the model as `[voice note not transcribed: ]` rather than silently vanishing. Spoken words are steering, never control: commands are parsed from typed text only (invariant 32). Each heard note is a `voice_transcript` activity on the ledger. There is no separate batch transcription endpoint. **The socket protocol** (`crates/vak-voice/src/protocol.rs`, version 1). Binary frames are 16 kHz mono 16-bit PCM. Client and server controls are separate types: a client sends only `speech_started`, `speech_stopped` and `playback`; the server sends `ready` (with a required `protocol_version`), `transcript`, `discarded` (`insufficient_speech`, `empty_transcript`, `rate_limited`), `turn_completed` and `error`. A transcript is therefore something only the server's governed route can produce — a client frame claiming to be one fails to decode and closes the session. At most one utterance is open; ids are never reused; audio outside an open utterance is dropped; stopping voice mid-sentence sends no `speech_stopped`, so the partial audio is discarded rather than submitted. A final transcript is appended to the session ledger and started as an ordinary governed turn; its answer returns as `turn_completed` with the same utterance id, and the client's `playback` frame for that id is appended as a `voice_playback` activity recording what was actually emitted. **Speech evidence, measured by the server.** A fixed loudness threshold cannot tell a person from a room: in a noisy room every frame clears it, which is how the 2026-09-23 open-mic run dispatched unrelated turns. `vak_voice::vad::SpeechEvidence` (`crates/vak-voice/src/vad.rs`) measures each closed utterance against its own quiet floor (the 20th-percentile frame level); a frame is voiced only when it stands 2.5× (about 8 dB) above that floor and above an absolute minimum, and an utterance needs 240 ms of voiced audio before it may cost a provider call. Flat noise of any loudness is discarded with `insufficient_speech`. The browser detector (`crates/vak-client-ui/src/voice-activity.ts`) applies the same constants live for endpointing — calibrating its floor from the quietest moment of the first 300 ms, adapting it while idle, learning the room's level from any utterance that turned out to be noise, capping an utterance at 30 s, and doubling the level and duration an onset needs while an answer is playing so the answer's own echo does not interrupt it. A test pins the shared constants. The client shapes latency only; the server decides. **Local engines.** `VAK_LOCAL_TRANSCRIBER` names an executable that reads encoded audio (WAV from the socket) on stdin and prints the transcript; `VAK_LOCAL_TTS` names one invoked with the requested encoding (`wav`, `pcm16`, `ogg_opus`, `mp3`) as its only argument, reading text on stdin and writing that encoding. Both run with an empty environment and are resolved through `vak_config::get_var`; an unconfigured engine fails closed. **Doctor** reports voice as a failure with a remedy when it is enabled but has no valid provider, no credential for it, or no model pin. **Evidence.** `crates/vak-server/tests/voice_session.rs` drives a real WebSocket upgrade against the real router with a scripted transcriber and model: flat noise is discarded without reaching the transcriber, speech becomes one Agent turn whose answer returns on the socket, playback is in the ledger, a forged client transcript is refused and never logged, a cross-origin upgrade is refused, and a session with no provider route is refused at connect. `crates/vak-server/tests/voice_channels.rs` posts a voice note through the gateway: a pending chat's note never reaches the transcriber, and an admitted chat's note reaches the model as its words. **Open.** A human-heard, physical-microphone round trip in a quiet room; a live channel voice note through a real bot; realtime streaming adapters; retiring the separate one-shot narration path in the client once spoken conversation covers it. Credential handling is intentionally outside the voice protocol: provider keys are resolved through vak's existing secret lookup chain and are never included in control frames, session transcripts, provider descriptors, or health/admin responses. Related: 38-voice-personality (the narration path this replaces and the `VoiceConfig` inheritance tier it keeps), 41-capability-registry (the visibility failure class this reproduces and must close), 48-web-client (the host seam and WebSocket precedent this reuses), 31-network-resilience (the four-plane network contract this must respect), 22-gateway (the Telegram media-I/O gap this closes), 01-llm (`Provider`, `ProviderAuth`, `ProviderRegistry`, receipts — the abstraction voice was built outside of and now joins). --- ## Problem (historical baseline) The following describes the gaps that motivated this design. It is retained to explain the implementation decisions; statements about the old narration path and missing diagnostics describe the pre voice-foundation release and must not be read as the current product status. Voice shipped once (`1380497 feat: voice & personality via Gemini Live`) and is present in the committed bundles, but it is invisible for six independent reasons — and even fully repaired it would not be what "live voice" means, because **there is no microphone code anywhere in the repo** (`getUserMedia`, `MediaRecorder`, `AudioWorklet`, WebRTC, any STT: zero hits). Doc 38 lists full-duplex as an explicit non-goal. 1. **It is narration, not conversation.** What ships is one-shot text→WAV that says `"Done."` / `"Task failed."` (`crates/vak-client-ui/src/store.ts`) and reads permission prompts aloud (`crates/vak-client-ui/src/components/ChatPane.tsx`). Two callers, both cues. 2. **Off by default, buried.** `voiceEnabled: false` (`store.ts:212`), in a group under Settings → General (`crates/vak-client-ui/src/components/Settings.tsx`). 3. **Silent failure without a Gemini key.** `/voice/speak` 400s (`crates/vak-server/src/lib.rs`); the client swallows it into `console.error` (`store.ts:364`). Flip the toggle, hear nothing, learn nothing. 4. **On desktop it is CSP-blocked even with a key.** `speak()` plays a `blob:` URL, but `crates/vak-desktop/tauri.conf.json` declares no `media-src`, so it falls back to `default-src 'self'`. `img-src` was given `blob:` explicitly; media was missed. There is also no `NSMicrophoneUsageDescription`, and no `ws:` in `connect-src`. 5. **Invisible to every diagnostic.** No `doctor` check, no `/providers` row, no capability entry. Doc 41 names this exact failure class — "the person who could repair the configuration saw nothing" — as the thing this project set out to kill. Voice reproduces it. 6. **The documented Telegram path was never built.** Doc 38 claims the gateway attaches a voice note when a chat resolves to a voice. The bridge downloads bounded voice notes, transcribes through the governed endpoint, and converts synthesized output to Telegram-compatible Ogg/Opus for `sendVoice`; failures preserve the text reply. Structurally, the root cause of 3–6 was one thing: `google_live::speak()` is a **free function called straight from an axum handler**, not an impl of `Provider` (`crates/vak-llm/src/lib.rs`). It is absent from `ProviderRegistry`, from the circuit breaker, from the route ladder, from key management, and from `doctor`. The current implementation routes through the configured provider, requires an explicit discovered model where the provider needs one, and obtains voice identifiers from capability discovery. --- ## The design decision that shapes everything **vak's agent loop stays the brain. Realtime providers are used as streaming transcription and streaming synthesis, never as the conversational agent itself.** Gemini Live and OpenAI Realtime are conversational agents by default: stream audio in, *they* decide the turn ended, *they* generate the reply. Wiring that up directly would put a second, un-audited agent in front of the user — no session entries, no permission engine, no capability registry, no frozen route ladder, no receipts. That is a direct breach of invariant 1 ("model-visible means logged") and of the whole premise of doc 41. A voice turn must be the *same* turn a typed prompt produces, with the same receipts. Both providers make this practical rather than merely correct in principle: each has a transcription-only mode — OpenAI's Realtime session opened with `?intent=transcription`, Gemini Live's `inputAudioTranscription` — that gives streaming ASR with server-side VAD and no model turn. That is the mode this design uses. `crates/vak-agent` remains the only thing that answers. The latency cost is small for this product specifically: vak is a coding agent whose turns take seconds of real tool work, so the agent loop, not the speech round-trip, is the long pole — but it is not zero. A real agent turn can run tool calls for 20–30 seconds, and a voice UI that goes silent for all of it reads as broken in a way text never does. **Spoken progress acknowledgement** ("Looking at the test suite…") off tool-start events, not just an outcome sentence at the end, is load-bearing here, not polish — see client design below. The payoff of keeping the agent loop as the only brain: one trait covers hosted and local providers identically, and local stops being a second-class citizen. Provider-as-brain (bridging vak's tools through a provider's own function-calling, every call still routed back through the permission broker) stays a legitimate future extension; the trait below is shaped so it can be added as a third implementation rather than a rewrite. --- ## Architecture ### 1. New crate: `crates/vak-voice` Add to `Cargo.toml` members and `[workspace.dependencies]`. Modelled on `vak-delivery`: deliberately thin dependencies (`vak-llm` for `LlmError`/`WorkReceipt`/`WorkPurpose`, `tokio`, `tokio-tungstenite`, `futures`, `serde`), **no `vak-server`, no `vak-core`**. It owns the audio primitives that have no business in `vak-llm`'s text-streaming vocabulary. ``` crates/vak-voice/src/ lib.rs VoiceError, VoiceRegistry, VoiceDescriptor, default_registry() audio.rs PCM framing, i16<->f32, resampling, WAV wrap/parse (absorbs wrap_wav from google_live.rs) vad.rs Energy + hangover endpointing for client-side turn-taking listen.rs Listener / ListenStream / ListenEvent — streaming ASR speak.rs Speaker / SpeakStream / synthesize_all() — streaming TTS transcribe.rs Transcriber — batch, container-in (Telegram voice notes) gemini.rs Gemini Live, transcription-only mode (absorbs google_live.rs) openai.rs OpenAI Realtime transcription session + /v1/audio/speech ``` `local` is not a fourth implementation module: it is the same OpenAI-compatible client (`openai.rs`) pointed at a different `base_url`, exactly the way `ollama` is just `openai`-shaped with a different default endpoint in `crates/vak-llm/src/registry.rs`. Because `ProviderAuth` already carries `base_url`, "local" needs no special case — `whisper.cpp --server` speaks the same `/v1/audio/transcriptions` shape OpenAI does, and Piper/Kokoro speak the same `/v1/audio/speech` shape. `crates/vak-llm/src/google_live.rs` **moves** into `crates/vak-voice` rather than being duplicated — it has exactly one caller today (`voice_speak`), so this is cheap, and it removes the free-function bypass around the provider abstraction described in the Problem section. `pub mod google_live;` comes out of `crates/vak-llm/src/lib.rs`, and the AGENTS.md crate map is updated in the same commit. ### 2. The trait set Three traits, because there are genuinely three shapes and collapsing them costs a fake shim in one direction or the other: streaming ASR, batch ASR (a Telegram voice note is a *file*, not a stream), and streaming TTS. A provider that is duplex (Gemini Live, OpenAI Realtime) is one struct implementing two of these traits; that is an implementation detail, not a fourth shape. ```rust /// What a provider says about itself, so no surface hardcodes a /// catalogue (invariant 9) — this is what deletes both the hardcoded /// Gemini model string and the hardcoded Settings.tsx voice list. pub struct VoiceDescriptor { pub name: &'static str, pub env_var: Option<&'static str>, pub default_base_url: Option<&'static str>, pub endpointing: Endpointing, // Server | Client pub formats: &'static [SpeakFormat], // Pcm16 | OggOpus | Mp3 pub voices: &'static [&'static str], // empty ⇒ discovered at runtime } #[async_trait::async_trait] pub trait Listener: Send + Sync { fn name(&self) -> &str; fn endpointing(&self) -> Endpointing; fn circuit_key(&self) -> String { self.name().into() } // mirrors Provider::circuit_key async fn open(&self, spec: ListenSpec, cancel: CancellationToken) -> Result, VoiceError>; } #[async_trait::async_trait] pub trait ListenStream: Send { async fn push(&mut self, frame: &[i16]) -> Result<(), VoiceError>; /// Client-endpointed engines only: "the utterance ended". async fn flush(&mut self) -> Result<(), VoiceError>; async fn next(&mut self) -> Option; /// Consumes the stream; returns the transcript accumulated so far so a /// cancelled listen still yields what the user said (invariant 5). async fn close(self: Box) -> Option; } pub enum ListenEvent { SpeechStarted, Partial { delta: String, text: String }, // delta AND snapshot (invariant 4) Final { text: String, confidence: Option }, SpeechStopped, Failed(VoiceError), } #[async_trait::async_trait] pub trait Speaker: Send + Sync { fn name(&self) -> &str; fn output_rate_hz(&self) -> u32; fn supports(&self, format: SpeakFormat) -> bool; async fn speak(&self, spec: SpeakSpec, cancel: CancellationToken) -> Result, VoiceError>; } #[async_trait::async_trait] pub trait SpeakStream: Send { async fn next(&mut self) -> Option, VoiceError>>; /// What has been emitted so far. On barge-in this is what the user /// actually heard, and it is what the ledger records. fn emitted_ms(&self) -> u64; } #[async_trait::async_trait] pub trait Transcriber: Send + Sync { fn name(&self) -> &str; /// A whole recording, container included. We do not decode Opus /// in-tree: every OpenAI-compatible /v1/audio/transcriptions endpoint /// and Gemini both take the container directly. async fn transcribe(&self, audio: AudioBlob, spec: ListenSpec, cancel: &CancellationToken) -> Result; } ``` `VoiceRegistry` mirrors `vak_llm::registry::ProviderRegistry` exactly: name→factory map, `ProviderAuth`-keyed cache where a different key or base URL is a different provider instance. **Cancellation (invariant 5).** `CancellationToken` threads through `open`/`speak`/`transcribe`, raced in `tokio::select!` exactly as `google_live.rs` already does. Two partial-output contracts follow: a cancelled listen returns the transcript heard so far, never silently nothing; a cancelled speak (barge-in) reports `emitted_ms()`, which is recorded as the truth about what the user actually heard — the alternative would let the ledger claim the agent said something the user never heard. **Receipts.** `WorkPurpose` gains `SpeechRecognition` (additive; `VoiceSynthesis` is unchanged). Every `Listener::open`, `Transcriber::transcribe`, and `Speaker::speak` produces a `WorkReceipt` through the same `record`/`classify_error`/`settle_cancelled` path `voice_speak` already uses. A duplex socket produces one receipt per utterance, not one per socket connection. ### 3. Transport: `WS /voice/session` Copy the shape of `pty_socket` (`crates/vak-server/src/web.rs`) — it is the established precedent for a bidirectional socket on this server, including its refusal gate. - **Frames.** Binary = raw mono i16 PCM, 20 ms frames. Text = JSON control (`{"t":"transcript",...}`, `{"t":"barge"}`, `{"t":"error","message","remedy"}`, …). Nothing a caller can say is mistakable for a control message, mirroring `PtyControl`. - **Rates.** The client's capture worklet is the only place the true device sample rate is known, so it opens the capture context at 16 kHz directly (Chrome/Firefox do the anti-aliased conversion in the audio thread) and downsamples in software only where the platform won't honour that. The server declares its output rate in the session's `ready` frame and never resamples; the client's playback graph resamples for free during output. - **Auth at upgrade** reuses the existing split: same-origin cookie on web, `?token=` on desktop. - **Refusal gate.** A `voice_refusal(&state, &headers)` mirroring `terminal_refusal`: off unless `[voice] enabled`. Deliberately **not** loopback-pinned the way the terminal is — the terminal's pin exists because a shell is the one effect not mediated by the permission engine, and a voice session only ever produces a *prompt*, which goes through the ordinary permission engine, hooks, broker and sandbox. What the socket does need, because it is an authenticated endpoint spending a paid per-minute API, is a concurrency cap (`max_concurrent`) and a wall-clock ceiling per session (`max_session_secs`) — the same reasoning `MAX_SPEAK_TEXT_CHARS` already documents for `/voice/speak`, generalized from a character cap to a session cap. - **A live security gap to close in the same change.** `crates/vak-server/src/lib.rs` computes `mutating` — whether the `Origin` header is checked — from the HTTP method, and a WebSocket upgrade is a GET, so `/pty` skips the Origin check entirely today. A `/voice/session` upgrade opens a mic relay that spends provider money; that is state-changing in every sense that matters even though it is a GET. `voice_refusal` must do its own `Origin` check, and the same fix belongs on `/pty` in the same change — shipping one checked WebSocket route and one unchecked one would be a worse inconsistency than either alone. - **Why relay through the server at all**, rather than letting the browser talk to Gemini/OpenAI directly with an ephemeral client token: the API key must never reach a webview (invariant 8), and the relay is where receipts, FinOps admission, and the session ledger attach. A browser-direct dispatch produces no `WorkReceipt` and no ledger entry. ### 4. Turn integration A final transcript is fed to the **existing** `sendPrompt` path — a voice turn is an ordinary turn with an ordinary session entry, marked with a `source: "voice"` field. The agent's streamed reply is chunked into `Speaker::speak()` at sentence boundaries so playback starts before the turn finishes. Speech detected during playback triggers an interrupt plus a playback-buffer flush; a mid-run barge-in also invokes the existing `cancel_run`. The modality gate is inert today only for want of one branch. `Modality::Audio` exists (`crates/vak-intent/src/axes.rs`), `leg_supports_modalities` already gates on it, and `crates/vak-intent/src/signals.rs` already folds any attachment's modality into `Reading::input_modalities` generically — but `crates/vak-core/src/lib.rs` is the only place an `Attachment` is ever constructed, and it hardcodes `Modality::Image`. Adding the audio branch there brings the whole gate alive with no further plumbing. ### 5. Client (`crates/vak-client-ui`, SolidJS) **Ship the browser first, desktop second — not the reverse.** The Tauri webview is the one host where microphone access is genuinely unproven: wry 0.55.1's WKWebView delegate unconditionally grants `getUserMedia` (`wry_web_view_ui_delegate.rs:126`), so the webview layer is not the risk, but `getUserMedia` also requires a secure context and the desktop origin is `tauri://localhost` — a custom scheme, not `http://localhost` — and whether WebKit treats that as potentially-trustworthy cannot be settled by reading code. That is a half-day spike that must run before any desktop voice work, not discovered partway through it. **Browser (Phase 3):** - New `HostFeature: "microphone"` in `host/port.ts`, plus a `VoiceTransport` interface modelled on the existing `TerminalTransport`. A host that cannot capture renders differently, not brokenly — the rule the file already enforces. - **Capture and playback are `AudioWorklet` modules shipped as real files** (not inline/blob) — required so the desktop CSP's `script-src` need not be widened to `blob:`. Capture downsamples to 16 kHz and frames to 20 ms; playback is a ring buffer fed from WS binary frames, explicitly **not** `