A real-time voice AI agent with retrieval, a cloned voice and a video avatar

The product records a person's own audio and lets specific people they've nominated ask questions about it later, and hear the answers spoken back in that person's voice. Four requirements fell out of that, and each one rules out the easy version of the system.

The pipeline is deliberately VAD → turn → STT → retrieval + LLM → TTS, not an end-to-end speech-to-speech model. Two things force text into the middle: retrieval needs a text query to embed, and the injection guard needs text to inspect. An end-to-end model would be fewer moving parts and strictly less controllable: you can't filter what you never render as text.

This is the design decision I'd defend hardest. At ingest, every Qdrant point is stamped with the owner, the source type, the list of people it was assigned to, and whether it's visible to everyone. At query time the caller is classified: owner or nominated user, and that becomes a hard filter on the search itself.

The consequence is that unauthorised content is never retrieved, so it can never reach the prompt, so no amount of clever phrasing can talk the model into repeating it. Filtering after retrieval, or trusting the system prompt to enforce a rule, would both have been less code and a genuine hole. It does mean a schema change has a migration problem: pre-existing vectors had no payload and were invisible until a backfill stamped them, which is the right failure direction, invisible rather than over-shared.

The parts have completely different load profiles, so they're separate deployments on K3s with their own node roles: the API and workers, the LiveKit SFU, the voice agent consumers, and the autoscaler as its own process. The GPU work (streaming speech-to-text, synthesis, the LLM, the embedder, the avatar renderer) runs on rented GPU pods and serverless workers I operate, not on vendor model APIs. Postgres holds the text and the job state, Qdrant holds the vectors, Redis carries the queues, and object storage holds the media behind presigned URLs.

Measured on a real call, from turn-final (the moment speech-to-text decides you've stopped) to the bot being audible is 0.83–1.17s. From when you actually stop talking it's 1.3–2.1s, and the difference is a deliberate 0.4–0.8s debounce before finalising the turn.

That debounce is the most interesting trade-off in the whole system. Shorten it and the agent cuts people off mid-sentence, which is far more irritating than waiting; lengthen it and the conversation drags. It is a product decision disguised as a timeout.

Generation is routed pod-first with overflow to serverless: always-on GPU pods take the traffic, and serverless workers are warmed before the pods saturate rather than after, because provisioning is not instant. Each model can be backed by a pool of pods, selected least-utilised-first, with a per-endpoint circuit breaker so a dead pod costs one fast failover instead of every subsequent request.

The piece I'm most pleased with is that the autoscaler learns capacity rather than being told it. It starts from a configured concurrency limit, watches for the point where a pod actually saturates (queuing, or per-request decode rate falling below a floor), clamps there, probes back up while healthy, and persists what it learned. That's what lets a pool mix GPU types without someone maintaining a threshold per type.

Capacity is measured, not guessed: one pod class benchmarks at ~11.5 requests/sec and ~25 concurrent users at p95 ≤ 6s, two pods at ~23 rps and ~50 users, so scaling is linear per pod. With a closed-loop load generator the useful law is just response time = concurrent users / throughput, which makes "how many pods for N users" arithmetic instead of an argument.

A video call is a voice call plus one more participant in the room. That framing came out of a constraint: the agent's transport couldn't publish video at all, so the renderer has to join the call itself rather than being a stage in the audio pipeline.

Cloning is zero-shot per request: the synthesis server is handed a reference clip and reproduces the voice, with no enrollment step or profile of our own to maintain. At call setup the agent picks the persona owner's most recent uploaded sample: the owner, not the caller, since the owner is who the call represents.

The two lessons were about caching and limits. The reference cache is keyed by the upload, not by the person: key it by person and a newer upload keeps serving the old voice forever. And the model hard-caps a reference clip at 30 seconds: an over-length sample makes every turn of a call fail with no audio at all, since the same rejected reference is reused for the whole call. That's a validation gap I'd rather fix by capping the recording UI upstream than by patching around downstream.

Consent for a cloned voice is owned by the layer where the upload happens, not here. That is the correct place for it, but it's worth being explicit that "the model can clone any voice from 30 seconds" is a product-policy problem long before it's a technical one.

Nearly all of the hard work was in the seams. Every individual model is a download with a documented interface; the system is hard because a turn crosses five of them plus a vector search, each stage can stall independently, and the user perceives the sum as one pause.

It's also the project where the AI work and the infrastructure work stopped being separable for me. Deciding to warm a serverless worker before saturation, or to key a cache by upload instead of by user, or to enforce permissions in a vector filter: none of those are model problems, and all of them decide whether the product works.

Client work under NDA, so there's no repository link and the product isn't named. Everything above is architecture and engineering reasoning, with measured numbers from my own load tests and instrumented calls: no product data, no client identifiers, no code.