Cutting a game's voice loop from seven seconds to one and a half

The character is a spectator. You place a building, the weather turns, you say something out loud, and she reacts: curious about what you're doing, with no idea what a level or a gold count is. That premise puts two hard constraints on the backend.

The first is latency. A companion who takes seven seconds to respond isn't a companion, she's a loading screen. The first working version did exactly that, and it was unusable in a way no amount of prompt tuning would fix.

The second is cost. A game fires events constantly: UI updates, stat changes, weather, buildings appearing. If every event became an inference call and a text-to-speech call, the per-player bill would scale with how much the player fidgeted.

The original design was sequential and file-based: upload the audio, transcribe it, call the model, wait for the whole reply, synthesise the whole reply, write an MP3, return a URL, let the client download it. Every stage waited for the one before it to finish completely, and the player waited for all of them.

Getting to ~1.5s took six changes, and only one of them was about making a component faster.

The endpoint returns a chunked stream rather than a finished file. Synthesis begins on the first sentence, so the player hears the start of the answer while the model is still producing the end of it. This is the single biggest win, and it's a change in the shape of the response, not in the speed of anything.

Even a fast pipeline has an unavoidable floor: transcription, plus the synthesis handshake. So the stream opens with a pre-loaded audio file (a soft breath, held in memory at startup) and the real reply follows behind it.

I like this one because it isn't an engineering fix at all. The measured pipeline is the same length; the perceived wait drops to nearly zero because the character audibly reacts the instant you stop talking. In a product where the character is meant to feel present, a breath is a better answer than a faster GPU. It also has to be content-neutral: it plays before the system knows whether you said hello or asked a question, so it can't be anything specific.

Transcription is a blocking GPU call, and the naive version stalled the whole async service while it ran: including the filler audio that was supposed to be covering it. It runs in a thread now, kicked off as a task while context and memory are assembled, so the pieces that don't depend on each other stop pretending they do.

The character's personality, behavioural rules and examples were roughly 80% of the tokens on every single request, and they never change. They're now loaded once at startup and held as a static system prompt, while the per-turn layer carries only what actually varies: time of day, recent observations, recent conversation, the classified event, and what the player just said. Replies are capped at 80 tokens, which keeps her to one or two sentences: shorter is both cheaper and, for this character, better writing.

This one was invisible until I measured it. Time-to-first-byte includes the player's own upload, and a WAV from a phone microphone is roughly ten times the size of an equivalent MP3. I wrote a small benchmark that runs the same request in both formats and reports the difference, then changed the client recommendation. No server-side optimisation would have found this, because the time wasn't being spent on the server.

A streaming response that a reverse proxy decides to buffer is just a slow non-streaming response, and it looks identical to a backend problem from the client side. Explicitly disabling buffering on the response is a one-line fix for a bug that will otherwise waste an afternoon.

The game wants to tell the backend everything that happens. Most of it doesn't deserve a spoken response: a stat change or a UI toggle isn't something a character should comment on, and answering it is both expensive and annoying.

So events are classified before anything expensive happens. Buildings appearing or being destroyed, weather changing, world events, direct conversation and greetings get a reply; minor updates and idle states are absorbed silently. That removed 40–60% of inference calls, and it made the character better rather than worse: she comments when something worth commenting on happens.

Two smaller versions of the same idea: a dedicated endpoint lets the game feed observations in with no response at all, and synthesised audio is cached by a hash of the text and voice so repeated lines cost nothing the second time.

A character who "sees" your world will happily invent things she has never been told about. Early on she'd congratulate players on buildings that didn't exist, which is worse than saying nothing: it breaks the illusion that she's actually watching.

Two decisions came out of what the Unity side could actually do. The audio stream carries only audio, so the text for the on-screen bubble is fetched separately by a session id returned in the response headers: the alternative was multiplexing text and audio in one stream, which is more elegant and much harder to consume in a game engine.

And the world id lives on the client. The game generates one, stores it, and sends it with every request; the backend keys memory off it. That kept the whole first version free of a user-to-world mapping problem, at the cost of trusting the client with an identifier: the right trade at that stage, and the first thing I'd revisit for a real launch.

Client work, so the repository isn't public and the game isn't named. The numbers above are from my own instrumentation and benchmark harness: every stage of the request logs its own timing, which is the only reason it was possible to find where the seven seconds actually went.