Serving language models: vLLM and the layer around it

On the voice agent platform nothing in the critical path is a vendor model API. Streaming speech-to-text, generation, speech synthesis, embeddings and the avatar renderer all run on GPU pods and serverless workers I deploy and operate. The reason is control: the audio never leaves the environment, the cloned voices stay with the product, and nothing critical can be rate-limited or deprecated by someone else's roadmap.

What that buys in control you pay for in operations. Five model families want GPU at different points in the same turn, so capacity has to be budgeted per model rather than handed to whichever process started first, cold starts have to happen before traffic rather than during it, and the co-located embedder taught me a lesson the hard way: loading it at the same time as the LLM poisoned it into returning all-NaN vectors. It now waits for the model server's health check, self-tests its own output at load, and exits so the pod restarts rather than silently serving garbage that looks like HTTP 200.

That's the shape of most self-hosting work: not the serving itself, but the failure modes that don't announce themselves.

Case study: the real-time voice agent platform

Always-on GPU pods are cheaper per request and predictable; serverless workers are elastic but slow to provision. Running both, with pods taking traffic first and serverless absorbing overflow, gets most of each: provided you warm the serverless side before the pods saturate rather than after, because by the time you need it, starting it is already too late.

The number I could never get right by hand was per-pod concurrency. Set it too low and you pay for idle GPU; too high and latency collapses at the exact moment traffic arrives. And the right value differs per GPU type, so a mixed pool means maintaining a table of thresholds that is wrong the day someone adds a different card.

So the autoscaler measures it. It starts from a configured limit, watches for the point where a pod actually saturates (requests queuing, or per-request decode rate falling below a floor), clamps there, probes back up while the pod is healthy, and persists what it learned. A pool can then mix GPU types with no per-type configuration at all.

The rest is the unglamorous part that makes autoscaling survive real traffic: a peak-held warm floor so a burst can't reset it before capacity provisions, clamped scale-downs during cooldown, two loop cadences (fast polling of my own metrics, slower reconciliation against the provider API), and an observe-only mode in non-production so a dev environment can watch itself without spending money.

Load testing turned "how many GPUs do we need" from an argument into a calculation. One pod class benchmarked at ~11.5 requests/sec and ~25 concurrent users at p95 ≤ 6s; two pods at ~23 rps and ~50 users: linear per pod. With a closed-loop generator the governing relationship is just response time = concurrent users / throughput, which means a latency target and a user count give you a pod count directly.

The corollary is that a p95 number without the offered load beside it is meaningless, and a surprising amount of published GPU benchmarking is exactly that.

The naive way to serve a model is one request at a time. The GPU sits idle between tokens, throughput collapses under concurrency, and every additional user waits behind the one in front. vLLM was built to fix that, and two of its ideas do most of the work:

In the AI companion backend, vLLM is one upstream among several: the FastAPI service also calls speech-to-text and text-to-speech, and reads and writes session state. Keeping the model behind its own interface is what makes that manageable: the application owns the turn, the model server owns generation, and neither needs to understand the other's internals.

Everything except inference scales by adding replicas. Inference scales by adding expensive hardware, so the honest capacity number for the whole product is whatever the model server can absorb. Sizing the rest of the system beyond that just moves the queue.

Generation time varies with output length, which means the tail is long by nature. The application has to decide what happens when a turn takes too long, and that decision is product-specific: in a voice loop, a late reply can be worse than a short one.

A single end-to-end latency metric hides everything interesting. Time spent queued versus generating, tokens produced, concurrent sequences in flight: those are the numbers that tell you whether you need a bigger card or a smaller prompt.

A model server is a stateful, expensive, slow-to-start dependency that holds a large cache in memory and degrades under concurrency rather than falling over. That description fits plenty of things I've had to run properly on Kubernetes, and the same instincts apply: resource limits that reflect what the process actually consumes, rollouts treated as carefully as any stateful upgrade, and the OpenTelemetry, Prometheus and Grafana stack I already run for platform metrics pointed at inference instead.

The difference GPUs make is that capacity stops being elastic. You can't paper over a bad placement decision by adding replicas, so scheduling and VRAM accounting have to be right the first time.

The Kubernetes side of my work