Evals: if it can't block a deploy, it isn't an eval

The team finishes the RAG. Someone puts together an eval: forty questions, a large model that scores every answer from 1 to 5 with the prompt "rate the quality of this answer", an average. It comes out to 4.3. It ships.

That 4.3 authorizes nothing. It is a satisfaction survey that fills itself out: it has no unit, its effective range is two values, and it drags along biases that have been measured and published. An eval is something else — a regression test with an explicit failure criterion — and almost nothing the industry calls an eval qualifies.

What everyone does

The pattern is always the same. A set of prompts gets written by hand, usually the same day the system is finished and usually by the person who built it. A model is asked to act as judge with a 1-to-5 rubric. The scores get averaged. A dashboard goes up with faithfulness: 0.87, answer_relevancy: 0.91, context_precision: 0.84. And the decision to deploy gets made by looking at those three numbers.

This feels rigorous. It has decimals, it has jargon, it has a chart. And it measures nothing you can decide with.

Why it's wrong

First: an average over a model-generated Likert scale has no unit. 4.3 against 4.1 means nothing, because you don't know whether that difference comes from the system or from judge noise. Even at temperature=0 no provider guarantees you bit-for-bit determinism between runs — server-side batching changes results. The mechanism is boring and it is real: floating point addition isn't associative, (a+b)+c doesn't give exactly the same last bit as a+(b+c), and the order the kernel accumulates in depends on how it partitions the work, which depends on the batch size, which depends on how many users are hitting the model at that instant. You are comparing two numbers that move on their own.

Second: 1-5 rubrics collapse the range. Run the same rubric over a hundred answers and look at the histogram. Almost everything lands on 4 and 5. It is a twenty-minute experiment and you can reproduce it today. But you don't need to run it to know the result, and that is the interesting part: the judge has no judgment module. It has a probability distribution over the tokens it can emit after "Score:", and in the corpus it was trained on 8/10 and 4/5 show up orders of magnitude more often than 3/10 or 2/5. You are measuring token frequency, not answer quality. A metric whose effective range is two values cannot detect a regression in anything.

Third: the judge has biases that have been measured and published. The reference paper here is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023, arXiv:2306.05685). The good news: a strong judge reaches over 80% agreement with human evaluators, the same level of agreement two humans reach with each other. The bad news: that same paper documents position bias, verbosity bias and self-enhancement bias, and it measures one column almost nobody replicates — consistency, the percentage of times the judge returns the same verdict when you swap the order of the two answers. With the default prompt, GPT-4 holds its verdict around 65% of the time. One in three judgments flips because of the order you handed it the answers in. Large Language Models are not Fair Evaluators (arXiv:2305.17926) confirms it independently.

Fourth, and the one that really matters: the dataset is built from cases you already know work. The forty questions were written by the same person who wrote the prompt. They are availability-biased toward what the system does well.

An eval that has never failed isn't an eval. It's a screenshot.

The real problem

The problem isn't measurement. It is a binary decision under uncertainty: this change goes to production or it doesn't. That question has two possible answers and neither of them is 4.3.

An eval is a regression test: a versioned dataset, an explicit failure criterion, running in CI, with enough statistical power to detect the size of regression you care about. Nothing beyond that.

Anything that can't block a merge is telemetry, not an eval. Telemetry is fine — it just doesn't authorize you to deploy.

The dataset starts at the failures

An evaluation dataset isn't written. It's harvested. The cases come out of production logs, support tickets and the things that broke. Twenty cases that already blew up on you are worth more than two hundred synthetic ones, and synthetic cases generated by the same model you are evaluating are worse still: they inherit its distribution and test what the model already knows how to do.

Every case carries an input, a category and an explicit acceptance criterion. In YAML, versioned in the repo, next to the code:

- id: transf-monto-con-puntos
  categoria: extraccion_monto
  fuente: ticket-4412
  input: "mandale 1.250,50 a juan por el alquiler"
  espera:
    monto_centavos: 125050
    moneda: ARS
    prohibido: ["1250.50", "125050,00"]

- id: transf-monto-punto-miles
  categoria: extraccion_monto
  fuente: ticket-4551
  input: "transferile 1.500 a mi vieja"
  espera:
    monto_centavos: 150000
    moneda: ARS

- id: transf-monto-ambiguo
  categoria: rechazo
  fuente: prod-log-2026-05-14
  input: "pasale unos mangos a mi hermana"
  espera:
    accion: pedir_aclaracion
    tool_calls_prohibidas: ["ejecutar_transferencia"]

The second case looks trivial and it is the most expensive of them all. In Argentina 1.500 is one thousand five hundred: the period is the thousands separator and the comma is the decimal. A model trained mostly on the English convention reads 1.500 as one point five. The difference between those two readings is a factor of a thousand on a transfer. And there is no 1-to-5 rubric that catches it: the answer sounds impeccable, it is worded perfectly, and it is wrong by three orders of magnitude.

The third case is the other one that matters. A system that reads "unos mangos" — vague slang for "some cash" — as an amount and executes a transfer is an incident, not a 3 out of 5.

Assertions first, judge last

The rule is simple and almost nobody follows it: if you can verify it with code, don't use a judge. LLM-as-judge is the last resort, not the first.

In a payments system, most of what you care about is deterministically verifiable: the JSON validates against the schema, the amount in cents is exact, the tool that got called is the right one, the answer doesn't contain a CBU that wasn't in the context.

# evals/test_suite.py
from collections import defaultdict
import pytest, yaml
from app.agent import responder

CASOS = yaml.safe_load(open("evals/dataset.yaml"))
RESULTADOS = defaultdict(lambda: {"ok": 0, "total": 0})

@pytest.mark.parametrize("caso", CASOS, ids=lambda c: c["id"])
def test_caso(caso):
    cat = caso["categoria"]
    RESULTADOS[cat]["total"] += 1          # counted BEFORE anything can fail

    salida = responder(caso["input"])
    esperado = caso["espera"]

    if "monto_centavos" in esperado:
        assert salida.monto_centavos == esperado["monto_centavos"]
    for prohibida in esperado.get("tool_calls_prohibidas", []):
        assert prohibida not in [t.name for t in salida.tool_calls]
    for txt in esperado.get("prohibido", []):
        assert txt not in salida.texto

    RESULTADOS[cat]["ok"] += 1             # only if everything passed

The order of those two lines is the whole trick. If you count the case after the asserts, a failing case never makes it into the tally and the denominator shrinks on its own: the category that breaks the most is the one that reports the best pass rate. It is a silent bug that hands you pretty numbers, which is the worst kind of bug.

And separate retrieval from generation, because they get fixed differently. The retriever is measured with a dataset of (query, doc_id_correcto) and recall@k: it is deterministic, it runs in seconds and it costs nothing. If your recall@5 is 0.62, there is 38% of cases where the right document never reached the context. No prompt fixes that. You will spend two weeks polishing instructions to move a metric that is capped by your chunking.

The judge has to be evaluated too

For what's left — tone, whether the answer contradicts the context, whether it invented a commercial term — you do need a judge. With three non-negotiable conditions.

One: binary verdict, and the evidence before the verdict. One specific question, with an explicit criterion, returning JSON with the keys in an order that forces the model to cite first and conclude after.

JUEZ = """Vas a leer un CONTEXTO y una RESPUESTA.
Pregunta: ¿toda afirmación factual de la RESPUESTA está soportada por el CONTEXTO?

Devolvé JSON con estas claves EN ESTE ORDEN exacto:
{"afirmacion_no_soportada": "<cita textual de la RESPUESTA, o null>",
 "por_que": "<una oración>",
 "veredicto": "si" | "no"}

Una sola afirmación no soportada implica veredicto "no"."""

The order isn't cosmetic. The model generates left to right: if the verdict comes out first, everything after it is a rationalization of something it already said. If the quote comes out first, the verdict is conditioned on the evidence. And the quote is verifiable by code: if that string isn't literally in the answer, the verdict gets thrown out.

Two: the judge is a sample too, treat it like one. A judge call is a draw, not a measurement. Run every judgment three times and keep the majority — and send to human review the cases where the three votes don't agree, because those are exactly the ambiguous ones, and the ambiguous ones are what tell you which line of the rubric is badly written.

from collections import Counter

def juzgar(contexto, respuesta, n=3):
    votos = [_una_llamada(contexto, respuesta) for _ in range(n)]
    conteo = Counter(v["veredicto"] for v in votos)
    veredicto, apariciones = conteo.most_common(1)[0]
    return {
        "veredicto": veredicto,
        "unanime": apariciones == n,
        "a_revision_humana": apariciones < n,
        "votos": votos,
    }

Add the position check on top: run every comparison in both orders and discard the cases where the verdict flips. With the 65% consistency from the MT-Bench paper, this isn't paranoia — it is the only way to know whether you are measuring quality or measuring which answer went first.

Three: measure the judge against yourself. Label a hundred cases by hand and compute the agreement:

from sklearn.metrics import cohen_kappa_score, accuracy_score
print(accuracy_score(humano, juez), cohen_kappa_score(humano, juez))

If kappa is below 0.6, the metric you are reporting is the judge's opinion, not your business criterion.

How many cases: the part nobody does

This is the calculation that turns an eval into evidence. Your suite has 30 cases and 27 pass: 90%. The 95% Wilson confidence interval for that runs from 74.4% to 96.5%. With 200 cases and the same 90%, it runs from 85.1% to 93.4%.

And to detect a real drop from 90% to 85% with 80% statistical power you need around 690 cases per arm. To detect a drop from 90% to 80%, about 200.

The conclusion isn't "go build 700 cases". It is that with 40 cases you can only detect catastrophes — which is perfectly fine, as long as you say so. The sin is writing "we went from 87% to 91%" in the PR with n=45. That difference is noise and you are selling it as an improvement.

If you didn't compute the interval, you don't have a metric. You have an anecdote with decimals.

And never average across categories. The gate goes per category, with different thresholds: 100% where a failure costs money, 85% where it is an annoyance.

# evals/conftest.py
from evals.test_suite import RESULTADOS

UMBRALES = {"extraccion_monto": 1.00, "rechazo": 1.00, "tono": 0.85}

def pytest_sessionfinish(session, exitstatus):
    bloqueado = []
    ok_total = sum(r["ok"] for r in RESULTADOS.values())
    n_total = sum(r["total"] for r in RESULTADOS.values())
    for cat, r in sorted(RESULTADOS.items()):
        tasa = r["ok"] / r["total"]
        umbral = UMBRALES.get(cat, 0.85)
        if tasa < umbral:
            bloqueado.append(cat)
        print(f"{'GATE' if tasa < umbral else 'ok  '} {cat:20} "
              f"{r['ok']}/{r['total']} = {tasa:.0%} (umbral {umbral:.0%})")
    print(f"     {'global':20} {ok_total}/{n_total} = {ok_total/n_total:.0%}")
    session.exitstatus = 1 if bloqueado else exitstatus

And this is the scene that justifies the whole article:

ok   rechazo              12/12 = 100% (umbral 100%)
ok   tono                 38/40 =  95% (umbral 85%)
GATE extraccion_monto     17/20 =  85% (umbral 100%)
     global               67/72 =  93%
exit 1

93% overall. Green on the dashboard, green in the weekly report, and the merge blocked anyway — because three out of twenty amount extractions are wrong and each of those three is a transfer for the wrong number. A 93% average would have let you deploy that.

The eval that becomes the target

Here is the hole in this thesis, and it needs saying out loud: if the eval blocks the release, the eval becomes the target. Within three or four iterations someone will tune the prompt until the suite reports 96%, and the support complaint rate won't move a millimeter. It is Goodhart, and it arrives on its own.

The defense is cheap: reserve a holdout. Between 20% and 30% of the cases don't get looked at while iterating, don't get debugged, don't get used to explain why anything failed. They run once per release and get compared against the rest of the suite. If the gate says 96% and the holdout says 78%, you didn't improve the system — you memorized the dataset. Rotate the holdout every few releases, and when a holdout case gets contaminated because someone looked at it to fix a bug, it moves to the development set and never comes back.

What it costs and what you give up

Concrete numbers, with Anthropic API list prices as of August 9, 2026 — check the current ones before running the math with your own numbers, because this ages. Claude Haiku 4.5 is at USD 1 per million input tokens and USD 5 per million output.

A 300-case suite, with ~1,500 input tokens and ~200 output per judgment, run in both orders for the position check: 0.9 MTok of input and 0.12 MTok of output. USD 1.50 per full run. Add the judge's triple sampling and it is 4.50. Forty PRs in a month: USD 180. That is less than an hour of the team arguing about whether the prompt change improved anything.

The real trade-off isn't the money in tokens. It is three things.

• The labeling hours. It is the cost nobody budgets for and the only one that doesn't come down with model prices. Labeling a case by hand — read the input, read the context, decide the acceptance criterion — takes close to two minutes if you already know what the right answer was. A hundred cases is 3.3 hours: one afternoon. Four hundred cases is 13 hours from someone who knows the domain, not from an intern. And the 690 cases per arm from the power calculation is almost three full days of a senior person.
• Determinism. The Batch API gives you a 50% discount but it is asynchronous, so if you want the eval blocking the merge you pay full price. And a test that depends on a judge is flaky by construction. A flaky test ends up disabled within two sprints. That is why the judge has to cover the minority of the cases and the gate has to have margin over the threshold, not sit pinned to the limit.
• CI time. Every minute you add to the pipeline is paid by the whole team, all day long.

When not to do any of this

One concession, because this article is absolutist end to end and there is a case where it is wrong. Ask yourself what you do differently if the eval says 82% instead of 91%. If the honest answer is "nothing, we ship it anyway" — because it is a prototype, because the cost of an error is zero, because the user is the team — then don't build the suite. Write a runtime guardrail that rejects invalid output when it shows up, and move on. An eval that is never going to block anything is the same screenshot from the beginning, except now it cost you three days of labeling.

Measuring before believing doesn't mean having a number. It means having a criterion that can say no, with enough evidence behind it to stand by it when someone asks you why it isn't shipping today.