Evals are compressed production failures
An eval test case is not a test. It's a compressed record of a moment when the system failed, preserved to force the system to re-encounter that situation on every deploy.
Most engineers think of an eval as a test. You write a question, write the expected answer, run the system, check whether they match. That framing is technically accurate and practically incomplete.
A better way to think about it: an eval test case is a compressed record of a moment when your system failed, or nearly failed, preserved in a form that forces the system to re-encounter that situation on every deploy. It is institutional memory. It is a record of lessons your system has already paid the cost to learn.
The difference matters
If evals are just tests, you write them before you ship and mostly forget about them. If evals are compressed failures, you write them whenever something bad happens in production, and the set grows over time as the system encounters the world.
The second model changes your behavior in concrete ways. You log the query that confused the model last Tuesday. You log the hallucination a user flagged. You log the edge case your colleague found in testing. Each of these becomes an entry in the eval set. The eval set becomes a portrait of everything the system has ever gotten wrong, which is exactly the information you need to know whether a new version is better.
Without this, you are doing something more dangerous than shipping without tests. You are shipping without memory.
What makes a good eval case
The worst eval cases are synthetic questions generated by an LLM and checked by the same LLM. They tend to be easy, well-formed, and completely unrepresentative of what real users do. Real users ask ambiguous questions. They use domain-specific jargon. They expect context the system doesn't have. They phrase the same question five different ways and expect the same answer.
The best eval cases come from three sources. The first is real production queries: the actual questions users asked, anonymized. The second is adversarial cases: questions designed to find failure modes (ambiguous phrasing, contradictory evidence, missing context). The third is regression cases: things that broke once and can break again.
Start with twenty to fifty hand-curated cases. Run them after every significant change. When something breaks in production, add it to the set before you fix it. By the time you have two hundred cases, your eval set knows your system better than you do.
Separation is the design
The most important structural decision in an eval pipeline is separating what you measure at each layer. Retrieval quality and answer quality are different things. You can have excellent retrieval and poor generation. You can have poor retrieval and still produce a good answer because the model interpolated from parametric knowledge. Unless you measure each layer independently, you cannot tell which one is causing the failure.
Each one is a compressed failure like the kind this article means. Watch what a generic vendor quality score sees versus what actually broke.
| Layer | Metric | Tool | What failure looks like |
|---|---|---|---|
| Retrieval | Hit-rate@k — correct chunk in top-k | Manual labels or RAGAS context recall | Right answer, wrong source — model is guessing from parametric knowledge |
| Grounding | Faithfulness — answer claims supported by retrieved context | RAGAS faithfulness or LLM judge | Fluent answer that contradicts or ignores the retrieved documents |
| Answer quality | Relevancy — answer addresses the actual question | RAGAS answer relevancy | Correct facts, wrong question answered |
| Abstention | False negative rate — abstained when answer exists | Labeled eval set with known answers | System says 'I don't know' when sources contain the answer |
| Structure | Schema compliance — output matches required format | JSON schema validation / regex | Valid prose with malformed structure that breaks downstream consumers |
The gate that changes everything
The most underused eval practice is blocking deploys on eval regression. This is standard in software engineering: tests fail, build fails, nothing ships. For LLM systems it remains rare, because eval scores are probabilistic and teams don't know where to draw the line.
Draw it. Pick a threshold: faithfulness above 0.80, retrieval hit-rate above 0.75, no new failure categories in regression tests. If a change drops below the threshold, it does not ship. Not because you've achieved certainty, but because you've established a floor below which you will not go.
The reason automated metrics like these are trustworthy enough to gate on: Es et al. (2023), the RAGAS paper, report agreement with human annotators on WikiEval of 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance, high enough to replace manual review for day-to-day regression gating, but not high enough to skip human review entirely on high-stakes changes.
If the right chunk isn't appearing in top-5, no amount of generation quality will fix it.
The irony of evals is that the teams who need them most (the ones moving fast, shipping often, under pressure to iterate) are the ones most likely to skip them. But speed without evals is not speed. It is the illusion of speed, paid for later in production incidents and lost user trust that takes far longer to recover than any eval framework takes to build.
The strongest practitioners are saying the same thing about off-the-shelf eval metrics:
Evals are the most critical element of AI product development. But there are many misconceptions. Metrics promoted by the eval vendors, like "hallucination" or "toxicity," are ineffective and often miss domain-specific issues. No wonder 85% of AI initiatives fail (Gartner, Show more
References: Es et al., RAGAS (2023), DeepEval, LangSmith