I spent real years of my career around search internals – query pipelines, ranking, the machinery that turns a corpus into answers. So the first time I traced a retrieval-augmented system end to end, the diagram looked oddly familiar: parse, index, retrieve, rank, serve. A search engine, box for box, with one new box at the end where the results page used to be. Then I noticed what the new box changes. Search could fail in front of the user and be caught in a glance. This thing fails and keeps talking.
RAG is a search engine with one new reader. Everything that made search good transfers: freshness, ranking, query understanding. What changed is the failure mode – search failed loudly, with ten visibly wrong links; RAG fails quietly, because the model weaves whatever it was handed into confident prose.
- Two pipelines under every RAG system – one that runs before any question exists, one that runs on every question – and the search product they secretly are.
- Chunking as an editorial act, retrieval as a committee, and the reranker's second opinion.
- Five ways it breaks in production, each with the symptom that gives it away, and the two grades that catch them.
Stage 02 of the Route gives retrieval one paragraph and an instruction: build RAG over a corpus you care about, and make it cite its source. The previous dive settled what deserves the window. This one is about the machinery that produces the candidates – and about the ways that machinery fails once real users, real documents, and real deadlines arrive.
Two Pipelines, One Promise
Every RAG system is two pipelines wearing one acronym. The first runs before any question exists: it takes your documents apart and files them. The second runs on every question: it goes looking. Keeping them separate in your head untangles most RAG conversations, because they fail differently, scale differently, and are owned by different parts of the system.
Chunking Is an Editorial Act
The least glamorous step decides the most. Documents rarely fit an embedding whole, so they are cut into chunks – and every cut is a bet about what can be understood alone. Cut too small and the meaning scatters: a table loses its header, a step loses its warning, an answer loses its condition. Cut too large and the embedding dilutes – one vector trying to summarise five topics matches none of them well – and the winning chunk drags paragraphs of rent into a window you were trying to keep lean.
The craft is unglamorous and specific: split along the document's own structure – headings, sections, code blocks – rather than at arbitrary character counts; let neighbouring chunks overlap a little so a fact straddling a boundary survives on at least one side; and carry metadata with every chunk – source, section, date – because the pipeline downstream will need to filter, cite, and judge freshness. None of it is machine learning. All of it is editing.
Retrieval Is a Committee
Embeddings find meaning, and meaning is not enough. Dense retrieval will happily connect "the payment failed" to "the transaction declined" – and then fumble an exact part number, an error code, or a person's name, because tokens like KO-4417 carry identity, not semantics. Keyword search – the BM25 lineage that ran the web for decades – has exactly the opposite temperament: deadly precise on the literal, blind to the paraphrase. Production systems stopped choosing years into the era and now run hybrid: both retrievers vote, and a fusion step merges the ballots.
Above the committee sits the second opinion. Retrieval is built to be fast and a little generous – pull fifty candidates cheaply. The reranker is slower and smarter: a model that reads the question against each candidate and reorders them properly, so only the best handful spend window rent. Wide then narrow, cheap then careful – the same two-stage shape as every ranking system you have ever met. How the index produces fifty candidates from fifty million in milliseconds is its own machine, and the next dive.
Where It Breaks in Production
Every failure below is a retrieval failure that presents as a generation failure – the model takes the blame, the pipeline holds the cause. That is the quiet-failure problem in practice: the surface shows confident prose either way.
-
The stale index
The ingest lane ran once, at launch, and nobody scheduled it again. The policy changed, the price changed, the document changed – the vectors did not. Search's oldest production problem, freshness, transferred intact.
// the symptom: the right answer to last quarter's question. -
The retrieval miss
The answer exists in the corpus and did not make top-k. The model, handed nothing useful, answers anyway – from its training, from the vibe of the question, from nowhere. This is the RAG-shaped version of confidently wrong.
// the symptom: fluent, plausible, and citing nothing. -
The fragmented answer
Chunking split the fact: the rule in one chunk, its exception in another, and only one of them retrieved. The model completes the half it received with perfect confidence.
// the symptom: half a procedure, confidently finished. -
Similar but wrong
Embeddings measure semantic neighbourhood, not usefulness. The retrieved chunk is about the right topic and does not answer the question – close in meaning-space, useless in practice. Similar is not relevant; only a reranker or a grade can tell them apart.
// the symptom: the right topic, the wrong document, every time. -
The poisoned document
Whatever the pipeline retrieves, the model reads – including a document that carries instructions instead of information. The corpus is an attack surface, and ingest is the front door. Label provenance, and keep what the model may read separate from what it should obey.
// the symptom: an answer following orders nobody at the table gave.
Grade the Halves Separately
Because every failure above wears the same face at the surface, the only way to localise one is to grade the halves separately – the discipline of the evals dive, applied with a scalpel. Retrieval gets its own golden set: questions paired with the passages that should arrive, scored on whether they did. Generation gets the other half: given the right passages, did the answer actually use them – faithfully, with citations that point where they claim? When a RAG system disappoints, the first question is never why did it say that. It is which half failed – and with two grades, that question takes minutes.
Make It Cite Its Source
The Route's build order for this stage ends on three words that carry more weight than they look: make it cite its source. A results page was always an honest interface – ten links, take them or leave them, the user could see the evidence. The new reader took that transparency away, and citations are how you hand it back: every claim wearing a pointer the reader can check, every answer auditable back through the pipeline that produced it. The search engine under the answer never went away. Cite your sources, and the reader gets to see it working.
One box in the pipeline did all the finding. The next dive opens it: The Index Allowed to Be Wrong · How Vector Databases Work – the graph, the partitions, and the recall dial. Or step back onto the Route at Stage 02.