RAG systems decay silently
Three independent decay channels (document drift, query drift, model drift) degrade retrieval quality on different timelines. Most teams instrument none of them.
The failure mode that kills RAG systems is not dramatic. There's no crash, no error log, no spike on your dashboard. What happens is quieter: the system keeps answering questions, keeps sounding confident, and slowly, over weeks, the answers get a little worse. Then a little worse again. By the time users complain, you've been in production with a degraded system for a month.
This is not a retrieval bug. It is decay. And it has three completely independent causes.
Three things that decay, and they don't decay together
The first is document drift: your source material goes stale. Policies get updated, products change, documentation accumulates contradictions. The embedding index still contains the old chunks. The model still retrieves them confidently. The answers are grounded in evidence that was accurate six months ago.
The second is query drift: your users change. A product launch shifts what people ask about. A new team starts using the system. The queries that were well-represented in your original eval set stop matching the queries coming in from production. Your retrieval was tuned for a question distribution that no longer exists.
The third is model drift: your embedding model changes. Maybe you upgrade to a newer, better model. Maybe the provider updates silently. Either way, the embeddings of new documents live in a subtly different space than the embeddings of old documents. Your index is now a patchwork of two incompatible coordinate systems. Retrieval degrades in ways that look random because the geometry is broken.
Most teams instrument none of these. They monitor latency, cost, and error rate. They do not monitor whether their retrieval is actually finding the right things.
Same three decay channels as above, but with a playhead so you can see the order they actually fail in.
Measuring each channel
For document drift, the question is: what fraction of your retrieved chunks reference content that is still accurate? You can approximate this by tracking document age (simple but blunt), by running a small LLM-based freshness check on a sample of retrieved chunks, or, if you have structured data, by comparing retrieved entity attributes against a ground-truth database.
For query drift, you need to compare the embedding distribution of recent queries against a reference window of older queries. Maximum Mean Discrepancy with an RBF kernel is the right tool. Project your embeddings down to 50-100 dimensions with PCA first; high-dimensional distance metrics are noisy. Run the MMD test daily or weekly. When the score crosses your threshold, you don't necessarily have a problem yet, but you have a signal to investigate.
For model drift, the fix is simpler in theory and more expensive in practice: when you upgrade the embedding model, you must reindex everything. No partial migration. A mixed-model index is a broken index. Automate the reindex pipeline so it runs on any embedding model change without human intervention.
| Channel | Detection method | Alert trigger | Remediation |
|---|---|---|---|
| Document drift | Chunk age + spot-check freshness sample | Median chunk age > SLA threshold | Re-crawl and re-index stale documents |
| Query drift | MMD on rolling query embeddings vs. reference window | MMD score exceeds calibrated threshold | Expand eval set from new query distribution; re-tune retrieval |
| Model drift | Cosine similarity between old/new embeddings of same documents | Any embedding model upgrade event | Full re-index — no partial migration |
The canary you need
The simplest, most valuable monitoring you can add to any RAG system is a fixed eval set of 50-100 queries with known correct answers, run on a schedule: daily if traffic is high, weekly if it's lighter. Track retrieval hit-rate (did the right chunk appear in top-k?) and answer faithfulness (did the answer follow the retrieved content?) as a time series. You don't need a sophisticated framework to start. A cron job, a list of questions, and a spreadsheet is enough to see whether your system was better last Tuesday.
Fraction of eval queries where the correct chunk appears in top-5. The leading indicator of decay.
The teams that ship reliable RAG systems aren't using better models. They're treating retrieval quality as an ongoing operational concern rather than a deployment decision. The index is not an artifact. It's infrastructure. And infrastructure requires maintenance.
Even the vendors are signaling that the naive version of RAG has run its course:
RAG is dead. long live RAG — index, generate embeddings and start querying in just a few steps with the new @cloudflaredev autoRAG
References: RAPTOR (Sarthi et al., 2024), RAGAS, Pinecone