Most 'the model hallucinated' complaints in RAG systems are retrieval failures. Fix chunking, ranking, and grounding first — the generation step is rarely the weakest link.
Why: A chunk that carries its own context ('Refund policy > Enterprise plans') retrieves and reads correctly in isolation; a blind window does neither.
Why: Embeddings capture meaning but blur exact tokens; lexical search is the opposite. Production QA corpora consistently show hybrid+rerank beating either alone by large recall margins.
Why: If the right chunk isn't in the top-k, no prompt can save the answer. Separating the metrics tells you whether to fix the index or the prompt — they are different weeks of work.
Why: The blend is where confident wrong answers come from — stale prices, deprecated APIs, invented policy details delivered in a trustworthy tone with a real-looking citation.
Why: The two worst RAG incidents in practice are leaking a document to the wrong user and answering from a superseded version. Both are metadata problems, not model problems.
Why: Real user queries are unlike your test questions. The log is a free, perfectly representative eval set that compounds in value.
When a RAG system gives a wrong answer, instrumented pipelines show the failure lives in retrieval most of the time: the right passage was never fetched, was fetched but ranked below the cutoff, or was chopped so badly it no longer contained the fact. Generation — the model actually misreading a good chunk — is the minority case.
That ordering dictates the work: structure-aware chunking and hybrid retrieval with reranking are the highest-leverage changes, retrieval-only metrics tell you when you’re done, and grounding instructions plus citations contain whatever errors remain. Token cost matters here too — reranking lets you send 4 good chunks instead of 20 mediocre ones, which is also a direct saving (see the token optimization playbook).