Real-time voice AI: speech in, speech out
The loop
A spoken turn passes through four stages before the user hears anything, and the user experiences all four as one wait:
- Capture and transcribe. Audio comes off the client and Whisper turns it into text. This is where audio format, sample rate and how much you buffer before transcribing start to matter.
- Assemble the prompt. The transcript is combined with the character's persona and whatever session state is relevant to this turn. Cheap in compute, decisive in quality.
- Generate. The language model produces the reply. Served by vLLM, so multiple concurrent turns share one model server instead of queueing.
- Synthesise and return. Text-to-speech turns the reply into audio, and the updated session is written back.
Why it's harder than text chat
The budget is a sum
Nothing in the chain is independent. Transcription, generation and synthesis all land in the same silence the player is sitting through, so improving the model's speed by half buys nothing if synthesis owns the delay. The practical consequence is that each stage needs its own timing in your metrics: end-to-end latency alone can't tell you where to look.
Speech is a worse input than text
Transcripts arrive with filler words, homophones, names spelled however the model felt, and occasionally silence transcribed as something confident and wrong. The stage that consumes the transcript has to expect that. Prompting the model as if it received clean typed input is a reliable way to get strange behaviour on the days the microphone is bad.
Turn-taking is a design problem, not a bug
Deciding when the user has finished speaking is a decision the system makes, and both answers are wrong sometimes: cut too early and you truncate them, wait too long and the conversation feels sluggish. Where that boundary sits shapes the whole feel of the product.
Grounding adds a stage, not a footnote
As soon as the answer has to come from your documents rather than the model's memory, a retrieval step lands in the middle of the turn. In a chat window nobody notices it. In a call it competes for the same silence as everything else, and a transcription error becomes a retrieval miss becomes a wrong answer: the mistakes compound down the chain instead of staying local.
Cascaded, not end-to-end
It would be fewer moving parts to hand audio to a speech-to-speech model and skip the text in the middle. I keep the text on purpose: retrieval needs a query to embed, and the injection guard needs something to inspect. You cannot filter what you never render as text, and giving that up buys latency at the cost of control.
Don't let the model pick the language
On a multilingual deployment I watched replies drift into the wrong language mid-call, because a short transcript is weak evidence of what someone is speaking. The fix was to stop inferring it: the detected language code goes into the prompt explicitly, every turn.
The expensive stage doesn't scale like the rest
The web tier can be replicated for the cost of a container. Inference is GPU-bound and shared, so concurrency planning happens there, which is exactly the problem vLLM's continuous batching addresses, and why the serving layer deserves its own attention.
Where the second actually goes
Measured on a real call: turn-final to the bot being audible is 0.83–1.17s. From the moment the user actually stops talking it's 1.3–2.1s, and the gap is a deliberate 0.4–0.8s debounce before the turn is finalised. Quoting only the first number would flatter the system; the second is what a person experiences.
- Retrieval runs speculatively. The embed and vector search happen during the pause speech-to-text is already waiting out. It hits on essentially every turn and costs roughly nothing on the critical path: better than making retrieval faster would have been.
- Synthesis is sentence-pipelined. The first complete sentence starts synthesising while the rest of the reply is still generating.
- Batched synthesis serving halved first-chunk time: 770–960ms down to 450–500ms, and, more importantly, it's the part that keeps holding up as concurrent calls increase.
- Barge-in is the architecture, not a feature. Voice activity detection interrupts playback mid-sentence, so in-flight generation, queued synthesis and buffered audio all have to be abandonable on the spot.
Cloning the voice, and rendering a face
Cloning is zero-shot per request: the synthesis server takes a reference clip and reproduces the voice, no enrollment step to maintain. With a clone active, steady-state latency stayed close to the baseline; the surprise was that the first turn of a call pays a one-off 1.2–1.4s while the reference cache is cold, and that the cache key matters: keyed by person rather than by upload, a newer sample never takes effect.
The video side is a separate participant in the call rather than a stage in the audio pipeline, because the agent's transport can't publish video at all. It renders a face from a photo, lip-synced to the audio the voice path already produced: 12fps at 384×256 and about 67 seconds to first frame on a cold pod, which is why one call gets one renderer and video minutes are billed separately from voice.
A cloned voice or face is a consent question before it's a technical one. Where the upload happens is the right place to own that, but it's worth saying plainly that "30 seconds of audio is enough" is a product-policy problem, not a footnote.
The earlier one: an AI companion game
Before that I built the voice backend for a Unity game whose companion character reacts to the world you build: faster-whisper for speech-to-text, Llama 3.1 on vLLM co-located on the same GPU pod, and synthesis fed model tokens over a persistent WebSocket. It started at about seven seconds to first audio and ended at about one and a half: almost entirely by changing the shape of the response rather than the speed of any component.
It's also where I learned the trick I still like most: the stream opens with a pre-loaded breath while transcription finishes behind it. The measured pipeline is unchanged; the perceived wait is nearly gone.
Related
- LLM infrastructure & servingvLLM, batching and treating a model server as a dependency.
- AI engineeringThe wider view of how I build systems around models.
- AI companion voice backendThe real system: architecture, trade-offs and constraints.
- DevOps & observabilityInstrumenting a multi-stage pipeline so you can find the slow part.