Voice AI agents that hold a conversation

Strip the word "agent" of marketing and what you have is a state machine sitting on a media transport, deciding when it's your turn and when it's theirs. The chain is voice activity detection → turn finalisation → speech-to-text → the brain → speech synthesis, running inside a WebRTC session so audio flows both ways continuously rather than in request/response pairs.

The agent I built runs as a pipecat pipeline on a LiveKit transport, one pipeline instance per call, with worker processes pulling call jobs off a Redis queue and an admission limit that refuses new calls rather than degrading every call in flight. The brain is access-filtered retrieval plus an LLM. It doesn't call tools or plan multi-step tasks: it's a conversational agent, and being honest about that distinction matters, because the engineering problems below are specific to real-time speech and don't overlap much with tool-using agents.

Case study: the real-time voice agent

Something has to decide when you've finished speaking, and both wrong answers are bad. Cut too early and the agent talks over you mid-sentence, which people find genuinely rude. Wait too long and the conversation feels sluggish and slightly dead.

In my system there's a deliberate 0.4–0.8s debounce before a turn is finalised, and it's why the numbers differ depending on where you start the clock: 0.83–1.17s from turn-final to audible, but 1.3–2.1s from the moment the user actually stops talking. Reporting only the first number would be flattering and useless: the second is what the user experiences.

Letting someone interrupt is not a feature you add at the end. It means every stage (the in-flight generation, the queued synthesis, the audio already buffered for playback) has to be abandonable the instant voice activity is detected. An architecture that only cares about being fast will fight you here; one that treats a turn as cancellable work won't.

Five stages land in one pause. The most effective thing I did was move retrieval off the critical path entirely: the embed and vector search run speculatively during the pause STT is already waiting out. It hits on essentially every turn and costs roughly nothing, which is a much better outcome than making retrieval faster would have been.

The same instinct applies downstream: synthesise the first sentence while the rest of the reply is still being generated, rather than waiting for a complete answer.

Transcripts arrive with filler, misheard names and homophones. When that transcript is also your retrieval query, a transcription error becomes a retrieval miss becomes a confidently wrong answer. The failure doesn't stay in the stage where it happened, which is the single biggest difference from a text chatbot.

On a multilingual deployment I watched the model drift into the wrong reply language mid-call, because a short transcript is weak evidence. The fix was to stop inferring: the detected language code goes into the prompt explicitly, every turn.

Follow-ups are full of pronouns, so the retrieval query has to be contextualised from earlier turns or "what did she say about it?" retrieves nothing. History is token-budgeted rather than accumulated, and a cheap intent gate skips retrieval altogether for social filler and "say that shorter" requests: before any embedding happens, so it costs nothing when it's wrong in the safe direction.

A call isn't a request. It occupies a pipeline, a GPU pair for streaming STT and synthesis, and, on video calls, a whole renderer pod, for its entire duration. That changes how you scale: a call holds its GPU pair for the length of the call rather than reselecting per request, capacity is gated so admission fails loudly instead of every call degrading, and the binding constraint turned out to be streaming STT rather than synthesis, which only a load test would have told me.

How the serving and autoscaling layer works