ProjectConcept design
Eval-Gated Deploy: Blocking Releases on Regression
A release should re-encounter the failures that taught you something.
Documented
Why I made it
I designed this around a simple habit: when production teaches you a failure mode, preserve it so the next release has to face it too.
An eval test case is a compressed record of a moment the system failed, which means a deploy pipeline that doesn't re-run that set on every change is shipping blind. This is a design for a CI gate: run the production-failure-derived eval set (RAGAS faithfulness, retrieval hit-rate, schema compliance) before merge, block on regression, and pipe every new production failure caught by tracing straight back into the eval set as a new case.
Tools: RAGAS · CI/CD · Langfuse · Python
This is a design, not a shipped system. No code is public yet. It's the direct operational answer to two things I wrote about separately: see Evals are compressed production failures and Observability is the primitive, not a feature.
The problem
Most teams treat evals as something you write once, before shipping, and then mostly forget. That framing misses what an eval case actually is: a compressed record of a moment the system failed, kept around specifically so the system has to re-encounter that situation on every future change. A static eval set written on day one doesn't grow with the system; it stops representing what the system actually gets wrong within a few weeks of real traffic. A production-grounded eval set keeps the gate connected to reality.
Separately, most teams that do have an eval set don't gate deploys on it. Eval scores are probabilistic, thresholds feel arbitrary, and "block the release" is a harder sell than "we'll keep an eye on it."The thing that makes the sell possible is a threshold agreed before the failing run, when nobody is under pressure. Set after, it is negotiated down every time. The result is the same failure software engineering solved decades ago with CI: a regression ships, gets noticed in production, and costs far more to fix there than it would have in review.
How it would work
- The eval set grows from production, not just from launch planning. Every traced production failure (a flagged hallucination, a confusing query, an edge case a teammate found) becomes a new eval case, not just a bug ticket. This is the same idea as a regression test suite, applied to LLM behavior instead of code paths.
- Layered metrics, not one aggregate score. Retrieval hit-rate, answer faithfulness (via RAGAS), and schema compliance are tracked and gated separately. An aggregate score can mask which layer actually broke.
- The gate blocks, it doesn't just report. A PR that drops faithfulness below threshold, or fails any previously-passing regression case, does not merge; the same hard line a failing test suite already draws in this team's CI, just applied to eval scores instead of unit tests.
- Traces feed the eval set, closing the loop. Observability infrastructure (the same traces used to debug latency and cost) is the source of new eval cases: a production failure caught in a trace becomes a labeled case automatically proposed for the next eval run, not a one-off Slack thread.
| Metric | Gate threshold | Why this layer |
|---|---|---|
| Retrieval hit-rate@5 | ≥ 0.75 | If the right chunk isn't retrieved, no amount of generation quality fixes the answer |
| Answer faithfulness (RAGAS) | ≥ 0.80 | RAGAS faithfulness scoring shows 0.95 agreement with human annotators on WikiEval, high enough to gate day-to-day changes on, not high enough to skip human review on high-stakes ones |
| Regression pass rate | 100% | Every case that broke before must still pass; no exceptions for 'minor' regressions |
| Schema compliance | 100% | Structurally invalid output breaks downstream consumers regardless of how good the prose is |
Key design decisions
- Block, don't just dashboard. A metric that degrades on a dashboard nobody is required to look at before merging isn't a gate, it's a vanity chart. The threshold has to be a merge-blocking CI check, the same as a failing unit test.
- Source new cases from traces, not from a quarterly review. Eval sets written once and revisited occasionally drift out of sync with what the system actually does in production. Wiring trace-flagged failures directly into the eval backlog keeps the set current without relying on someone remembering to update it.
- Per-layer thresholds, not one number. A single blended score can pass while one specific layer (retrieval, grounding, format) is silently broken. Separate gates make it obvious which part of the pipeline regressed.
Status
Design only. The threshold table above is a reasonable starting point grounded in published RAGAS-human agreement numbers, not a tuned result from running this pipeline — those numbers would need to be calibrated against a real eval set and traffic pattern before they're more than a sensible default.
Limitations
- A gate is only as good as its eval set. If the eval set hasn't caught a failure mode yet, the gate can't block on it; this design reduces blind spots over time, it doesn't eliminate them on day one.
- Thresholds need calibration per system. The 0.75/0.80 starting points above are defensible defaults, not universal constants; a system with a harder retrieval problem may need a different bar.
- This is unbuilt. No CI integration, no real eval set, no measured false-positive rate on the gate itself (a gate that blocks too aggressively just gets bypassed under deadline pressure, which defeats the purpose).