RAG quality is a data problem first
The common mistake in retrieval-augmented generation is treating the vector database as the product. The product is the evidence path: what was retrieved, why, and whether the answer stayed faithful to it.
The common mistake in retrieval-augmented generation is treating the vector database as the product. The database is infrastructure. The product is the evidence path: what was retrieved, why it was retrieved, and whether the answer stayed faithful to it.
Retrieval-augmented generation works because it lets a model answer from a verifiable source instead of compressed memory: Lewis et al.'s original RAG paper measured a 10-point exact-match gap over a comparably-sized parametric-only model on Natural Questions (Lewis et al., 2020). That gap is the whole pitch. But the gap only holds if the evidence path is actually sound, and most production RAG failures trace back to the data pipeline that feeds it, not the generation model sitting at the end.
The retrieval stack has four failure points
When a RAG answer is bad, the visible failure usually looks like hallucination. The root cause often sits much earlier in the stack; by the time the model sees anything, the damage is already done.
| Stage | Failure mode | Fix |
|---|---|---|
| Ingestion | Important content is missing | Improve loaders and metadata coverage |
| Chunking | Context is split badly | Use semantic or document-aware chunks |
| Retrieval | Wrong evidence ranks high | Add reranking and query rewriting |
| Answering | Model ignores retrieved evidence | Add citations and refusal rules |
Barnett et al. audited production RAG systems and catalogued seven distinct failure points spanning exactly this range: missing content, chunks ranked too low to surface, content excluded after retrieval, and answers that extract the wrong information even when the right evidence was present in context (Barnett et al., 2024). None of those seven are generation-model problems. They're all upstream of the model ever seeing a token.
Retrieval and generation should be trained to expect each other
The default architecture treats retrieval and generation as two systems glued together: a retriever fetches passages, a generator reads them. Self-RAG argues this glue is itself a failure point: the generator has no signal for whether the retrieved passages are actually relevant or sufficient, so it generates regardless. Their fix trains the model to retrieve adaptively and critique its own output against the retrieved evidence, which improved factuality and citation accuracy over standard RAG and even some heavily-tuned proprietary systems on the tasks they tested (Asai et al., 2023).
The pattern across every gain in that chart is the same: nothing about the underlying generation model changed. What changed was whether the data reaching it was complete, well-chunked, correctly ranked, and checked against the final answer.
Engineering Implications
Treat retrieval quality as an operational metric, not a one-time architecture decision; it decays the same way any data pipeline decays if nobody is watching it.
Requires a labeled eval set — without one, this number doesn't exist.
The teams that ship reliable RAG systems treat the evidence path as the thing they're building, with the vector database as one component inside it: chunking, metadata, reranking, and citation-checking are the actual product surface, and a generation model swap rarely moves the needle compared to fixing any one of those.
References
- Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401
- Barnett, S., et al. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. arXiv:2401.05856
- Asai, A., et al. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv:2310.11511
- Es, S., et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217