AI engineering: the system around the model

The largest is a production voice agent: you call it, talk over it, and about a second after you stop you hear an answer in a cloned voice, from a face that lip-syncs to it, grounded in access-filtered retrieval rather than in the model's memory. I built the ingestion and retrieval pipeline, the agent, the serving layer and the autoscaler: every model running on GPUs I operated rather than behind a vendor API.

Before that, the voice backend for a Unity game whose companion character reacts to the world you build. It shipped at about 1.5s to first audio, down from about 7s, and the changes that got it there were streaming instead of finishing, masking the floor with a pre-loaded breath, and simply not calling the model for events that didn't deserve an answer, which removed 40–60% of inference calls.

Both are small systems by headcount and demanding ones by latency, which is a combination I like. Every architectural decision has an immediate, audible consequence.

A model server behind an HTTP API is just an upstream service with unusual performance characteristics: expensive, stateful in its cache, and much harder to scale horizontally than your web tier. Once you design it that way, the rest of the system gets normal: timeouts, retries, circuit breaking, capacity planning, all the usual tools apply.

In a multi-stage pipeline the end-to-end number tells you almost nothing about what to fix. Transcription, prompt assembly, generation and synthesis each need their own timing, because they degrade for different reasons, and the one that dominates changes with load.

Anything that remembers accumulates context, and unbounded context is both a cost problem and a quality problem. Choosing which slice of state, or which retrieved passages: matter for this turn is real design work, not a formatting detail.

Retrieval is what stops a model answering from memory about documents it has never seen. It also splits every wrong answer into two distinct bugs: the right passage wasn't retrieved, or it was retrieved and the model still got it wrong. Those have different fixes, and you can only tell them apart if you log what was retrieved.

How I build retrieval

Models are non-deterministic, occasionally wrong, and sometimes slow for no visible reason. That argues for more instrumentation than a conventional service, not less: structured logs of what went in, metrics per stage, and traces that follow a single turn end to end.

I came to AI work from infrastructure, and that has been an advantage rather than a detour. A model that has to serve real users needs the same things any production service needs: somewhere to run, a rollout process, resource limits that reflect what it actually consumes, and dashboards that tell you when it's degrading.