Retrieval-augmented generation, in production

A language model knows what was in its training data. It does not know your documents, and it cannot tell you that it doesn't know: it will produce something fluent either way. In the product I built this on, the content is a person's own recordings, so a confabulated answer isn't a quality issue, it's the worst thing the system can do.

Fine-tuning is the wrong tool here. It's expensive to repeat, slow to react when content changes, and it gives you no way to point at the passage an answer came from.

This is the part I'd defend hardest. Different callers are allowed to see different content, and the enforcement point is the vector search itself: at ingest every point is stamped with its owner, its type, who it was assigned to and whether it's visible to everyone, and at query time the caller is classified and that becomes a hard filter.

The reason to do it there rather than after retrieval, or worse, by asking the model nicely in a system prompt: is that content which is never retrieved can never reach the prompt, so no amount of clever phrasing can talk the model into repeating it. Prompt-level rules are not a security boundary.

It has a cost worth knowing about: when the payload schema arrived, existing vectors had no access data and were invisible to every query until a backfill stamped them. Invisible is the correct failure direction, but it is a migration you have to plan.

Given the wrong passages, a good model produces a confident wrong answer that sounds exactly as trustworthy as a correct one. Almost all the useful iteration happens in retrieval: what gets indexed, how it's split, what the query actually is, and almost none in rewording the prompt.

When an answer is wrong, the first question is whether the right passage was retrieved. If it wasn't, generation was never going to save it; if it was and the answer is still wrong, that's a different bug with a different fix. Logging what was retrieved per turn is what makes the distinction possible: without it you're tuning blind.

A greeting or an off-topic question should not be answered from an empty context block: that teaches the model to hallucinate structure. Sending the bare question when retrieval comes back empty, and letting the model answer naturally, was a small change that visibly improved behaviour.

"What did she say about that?" embeds to nothing useful. Contextualising the retrieval query from the last few turns is what makes multi-turn conversation work at all, and it's separate from putting history in the generation prompt: those are two different problems that both look like "add chat history".

Social filler and "say that again, shorter" don't need a vector search. A cheap heuristic gate decides before the embed happens, so it costs nothing in time-to-first-token: an earlier LLM-based version of the same gate blocked the first token behind its own model call, which is a lot of latency to spend on a decision that should be biased toward retrieving anyway.

Anything user-supplied that reaches a prompt is an injection surface, so there's a guard on the way in. It's also one of the reasons the voice pipeline converts speech to text rather than going speech-to-speech: you cannot inspect what you never render as text.

In a chat window, half a second of retrieval is invisible. In a spoken conversation it lands in the same silence as transcription, generation and synthesis, and the user has a much shorter fuse.

The fix that worked wasn't making retrieval faster: it was moving it off the critical path. The embed and search run speculatively during the pause that speech-to-text is already waiting out to decide the turn is over. It hits on essentially every turn and adds roughly nothing to the time the user perceives.

How the voice loop is put together