To evaluate an LLM application you need four things: a versioned set of real inputs with a written description of what a good answer looks like, a scorer for each input, a runner that executes your app on every case and records every score, and a threshold in CI that blocks a merge when the score drops. Everything below is detail on those four parts. With all four, you can change a prompt on a Friday. With none, the first user complaint is how you learn something broke.
Key Takeaways
- The eval set is the hard part. The runner is a weekend. Assembling 200 real labeled cases takes weeks, and it is the only thing that makes the number mean anything.
- Use a deterministic scorer wherever one exists. Schema validity, citation IDs, tool arguments, and banned strings are checkable in code. Save the judge for the fuzzy part.
- Calibrate the judge before trusting it. An uncalibrated judge is a random number generator with good grammar.
- Track two refusal errors. Over-refusal and under-refusal move in opposite directions, and one safety score hides the tradeoff.
- Expect offline and online to disagree. Measure the gap instead of arguing about which is right.
This assumes you already have something working: a retrieval pipeline, a support agent, a summarizer. The code is Python and uses the Anthropic SDK for the judge, but every pattern translates to any provider. Still building the thing? Start with the RAG pipeline guide or the Python AI agent walkthrough.
What Evaluating an LLM Application Actually Means
LLM evaluation is measuring whether your application produces useful output on inputs you care about, repeatedly, as the prompt and model change underneath it. It is not benchmarking the model. Leaderboards describe a model in isolation; your eval describes the model plus your prompt, retrieval, tools, and parsing code.
Three things make this unlike ordinary testing. Output is non-deterministic, so one passing run proves nothing. There is rarely a single correct answer, so equality assertions do not apply. And the input space is open natural language, so exhaustive coverage is impossible. Statistics replace all three. The unit of work is a run, not a test: it executes N cases and yields a rate with an interval around it, and two runs get compared rather than asserted.
The Four Parts, Concretely
- Cases: a JSONL file of inputs plus expectations, in git, reviewed like code
- Scorers: pure functions returning a pass, a fail, or a number
- Runner: executes every case with concurrency, caching, and full logging
- Gate: a CI threshold plus a diff of which cases changed verdict
How to Build an Eval Set
Start from real inputs, not invented ones. Pull 200 messages from your support queue, your logs, or your beta testers, cluster them by intent, and sample across clusters. Invented cases test the app you imagined; logged cases test the app you shipped.
The layout below keeps cases, scorers, and baselines in the repo so a prompt change and its eval change land in the same pull request.
evals/
cases/
faq.jsonl # 74 cases, general product questions
order_lookup.jsonl # 41 cases, requires a tool call
must_refuse.jsonl # 22 cases, out of policy
should_answer.jsonl # 31 cases that look risky but are fine
scorers/
deterministic.py
judge.py
run_eval.py
baselines/2026-07-14.json
prompts/
support_agent.v7.md
Each line of a case file is one case. Keep expectations machine-checkable where you can, and keep provenance so you can trace a case back to the incident that produced it.
{
"id": "faq-014",
"input": "do you ship to canada and how long",
"context_ids": ["kb/shipping-intl", "kb/delivery-times"],
"expect": {
"policy": "answer",
"must_contain": ["Canada"],
"must_not_contain": ["I don't have"],
"rubric": "States that Canada is served and gives a delivery window."
},
"source": "ticket-88213",
"added": "2026-06-02"
}
How many cases? Enough that the interval is narrower than the change you want to detect. Twenty-seven passes out of 30 reads as 90 percent, but the 95 percent Wilson interval spans 74.4 to 96.5 percent. The same 90 percent from 180 out of 200 spans 85.1 to 93.4 percent. A five-point regression is invisible at n=30 and obvious at n=200. Start at 30 to 50 per capability, then grow toward 150 to 300 before any number blocks a deploy.
Four rules keep a set healthy. Append cases rather than editing them, since editing rewrites history. Turn every production incident into a case the day it is fixed. Label every case with a policy (answer, refuse, escalate, clarify) so refusal behavior is measurable. And hold back roughly 15 percent for release checks only, because a set you optimize against hourly stops being a measurement.
Deterministic Scorers vs LLM-as-Judge
Reach for code first. Deterministic scorers are free, instant, perfectly repeatable, and never drift. A judge costs money, adds seconds, varies between runs, and changes behavior when the judge model is updated. Most teams reach for the judge far too early, because writing a rubric feels faster than writing a parser.
All of these are checkable in code with no model call: valid JSON against a schema, required fields present, a cited document ID appearing in the retrieved set, tool arguments in range, an answer inside a length band, a banned phrase absent, every number in the output present in the source, a label inside the allowed enum.
import json, re
from jsonschema import validate, ValidationError
def schema_valid(output: str, schema: dict) -> bool:
try:
validate(json.loads(output), schema)
return True
except (json.JSONDecodeError, ValidationError):
return False
def contains_all(output: str, needles: list) -> bool:
low = output.lower()
return all(n.lower() in low for n in needles)
def citations_in_context(output: str, allowed: list) -> bool:
# Answers cite sources as [kb/shipping-intl]
cited = set(re.findall(r"\[([a-z0-9/_-]+)\]", output))
return cited.issubset(set(allowed))
def numbers_traceable(output: str, context: str) -> bool:
# Every number in the answer must appear in the source text
nums = set(re.findall(r"\d[\d,.]*", output))
return all(n in context for n in nums)
That last one catches a surprising share of fabrication for zero cost. Reference metrics such as BLEU and ROUGE belong in a narrower box: they work when a reference answer exists and exact wording matters, mostly translation and tight extraction. On open-ended answers they punish good paraphrases, so treat a ROUGE delta as a smoke signal, not a verdict.
Fuzzy, context-bound qualities
Is this answer supported by the passage? Did it address the actual question? Is the tone right for a customer? Which of these two drafts would a reader prefer? These have no closed form, and a judge with the source material in front of it does well.
Domain facts it was never given
Is this dosage correct? Does this clause satisfy the regulation? Is this SQL semantically equivalent? A judge without the ground truth is guessing confidently. Supply the reference, run the query, or have a human label it.
How to Write an LLM Judge You Can Trust
A judge is a small, boring, single-purpose classifier. Give it one question, a structured output schema, and the evidence it needs. Do not ask it to rate quality from 1 to 10, because those scores cluster around 7 and move for no reason.
Structured output matters more than the rubric wording, because it removes the parsing step. That step is where a judge quietly loses results: a stray sentence before the JSON, a wrapped code fence, a verdict spelled a little differently than last week. A schema turns all of that into a field you can read. See the structured outputs guide for the mechanics.
from anthropic import Anthropic
from pydantic import BaseModel, Field
from typing import Literal
client = Anthropic()
JUDGE_SYSTEM = """You check whether an answer is supported by a source passage.
Split the answer into atomic factual claims. A claim is supported only if
the passage states it or directly entails it. General knowledge does not
count as support. Ignore style, tone, and length entirely.
Return "grounded" only if every claim is supported."""
class Groundedness(BaseModel):
verdict: Literal["grounded", "partial", "unsupported"]
unsupported_claims: list[str] = Field(
description="Verbatim claims the passage does not support.")
reason: str = Field(description="One sentence.")
def score_groundedness(answer: str, passage: str) -> Groundedness:
resp = client.messages.parse(
model="claude-opus-5",
max_tokens=1024,
system=JUDGE_SYSTEM,
messages=[{"role": "user", "content":
f"<passage>\n{passage}\n</passage>\n\n"
f"<answer>\n{answer}\n</answer>"}],
output_format=Groundedness,
)
return resp.parsed_output
Note what is missing: no temperature. Current Claude models removed the sampling parameters, and sending one returns a 400, so consistency comes from a tight rubric and a constrained schema rather than a dial. On providers that still expose temperature, pin it at zero and expect some variance anyway. Judging one claim against one passage is classification work, not research, so keep the judge cheap: a small model, a low reasoning setting, and a schema tight enough that there is nothing to ramble about.
Known judge biases
The MT-Bench work that popularized LLM-as-judge also documented its failure modes, and they have held up. Position bias: the judge favors whichever answer came first, so run every pair in both orders and treat a disagreement as a tie. Verbosity bias: longer answers score higher regardless of content, so normalize length or state that length is not a criterion. Self-preference: judges rate their own model family higher, so when comparing providers, judge with a third.
Calibrate before you trust
Label 50 to 100 outputs by hand, run the judge on the same outputs, and compute agreement beyond chance. Percent agreement alone is misleading when one class dominates.
def cohens_kappa(both_pass, human_only, judge_only, both_fail):
n = both_pass + human_only + judge_only + both_fail
po = (both_pass + both_fail) / n
pe = ((both_pass + human_only) * (both_pass + judge_only)
+ (judge_only + both_fail) * (human_only + both_fail)) / n**2
return (po - pe) / (1 - pe)
# 100 hand-labeled cases: 40 both pass, 5 human-only, 7 judge-only, 48 both fail
print(round(cohens_kappa(40, 5, 7, 48), 3)) # 0.759
A kappa around 0.76 is solid enough to gate on. Below roughly 0.6, read the disagreements: the rubric is almost always ambiguous rather than the judge being dumb, and one rewritten sentence usually fixes it. Re-run calibration whenever the judge model changes, and pin that version so a silent upgrade does not shift every historical score.
How to Regression-Test a Prompt Change
Treat a prompt like a source file. Version it, diff it, and never merge a change without a run against the previous baseline. The output of a prompt change review is not a score, it is a list of cases whose verdict flipped in each direction.
The runner does three things worth building carefully: it caches on a hash of prompt, input, and model, so a rerun after a scorer fix does not pay twice; it runs cases concurrently; and it writes every raw output to disk so you can read what happened.
import json, hashlib, pathlib
from concurrent.futures import ThreadPoolExecutor
CACHE = pathlib.Path(".eval_cache"); CACHE.mkdir(exist_ok=True)
def generate(case, prompt_text, model):
key = hashlib.sha256(
f"{prompt_text}|{case['input']}|{model}".encode()).hexdigest()[:16]
hit = CACHE / f"{key}.json"
if hit.exists():
return json.loads(hit.read_text())["output"]
output = your_app(case["input"], prompt_text, model) # your code
hit.write_text(json.dumps({"case": case["id"], "output": output}))
return output
def run(cases, prompt_text, model, scorers, workers=8):
def one(case):
out = generate(case, prompt_text, model)
scores = {name: fn(out, case) for name, fn in scorers.items()}
return {"id": case["id"], "output": out, "scores": scores,
"passed": all(scores.values())}
with ThreadPoolExecutor(max_workers=workers) as pool:
return list(pool.map(one, cases))
Now compare two runs. The headline pass rate is the least useful number in the report; what you want is the flip list and a test of whether the difference is real. For paired binary outcomes on the same cases, McNemar's test looks only at the cases that changed.
from math import comb, sqrt
def wilson(k, n, z=1.96):
p = k / n; d = 1 + z*z/n
c = (p + z*z/(2*n)) / d
h = z * sqrt(p*(1-p)/n + z*z/(4*n*n)) / d
return c - h, c + h
def mcnemar_exact(fixed, broken):
# fixed: failed before, passes now. broken: passed before, fails now.
n = fixed + broken
tail = sum(comb(n, i) for i in range(0, min(fixed, broken) + 1))
return min(1.0, 2 * tail / 2**n)
def compare(before, after):
b = {r["id"]: r["passed"] for r in before}
fixed = [r["id"] for r in after if r["passed"] and not b[r["id"]]]
broken = [r["id"] for r in after if not r["passed"] and b[r["id"]]]
return {"fixed": fixed, "broken": broken,
"p": mcnemar_exact(len(fixed), len(broken))}
# 12 fixed, 4 broken over 200 cases -> p = 0.0768, not yet conclusive
That last comment is the point of the section. Twelve fixed against four broken sounds like a clear win in standup, and it is not: the exact two-sided p-value is 0.077, so a run that size cannot rule out noise. Ship it anyway if the four are trivial and the twelve are user-visible, but ship knowing what the evidence supports. Read the broken list every time. A prompt edit that raises the aggregate while breaking your two highest-value cases is a bad trade a single number will hide.
How to Measure Hallucination and Refusal
You cannot measure truth, so measure groundedness instead: the share of answers where every claim is supported by the context you supplied. That is checkable, because the context is sitting right there. For a deeper treatment of why models fabricate, see the piece on AI hallucination.
Build it in two layers. The cheap layer runs first with no model call: a citation ID outside the retrieved set is an automatic fail, and so is a number absent from the source. Only survivors reach the judge, which cuts spend sharply and catches the worst failures deterministically.
Report groundedness separately from retrieval recall. An answer perfectly grounded in the wrong three documents is still wrong, and merging the metrics makes a retrieval bug look like a generation bug. If the index is the suspect, the vector database comparison covers that side.
Refusal is two error rates, not one
Safety-adjacent evals need two case files pulling in opposite directions. must_refuse.jsonl holds genuinely out-of-policy requests; the metric is under-refusal, the share answered anyway. should_answer.jsonl holds requests that merely pattern-match to something risky, such as medical vocabulary in a benign question; the metric is over-refusal. Report both. One "safety score" lets a change that hardened one side and wrecked the other look like progress.
# 1. Policy-level decline: HTTP 200, but the API tells you directly.
# Check this BEFORE reading response.content, which may be empty.
if response.stop_reason == "refusal":
category = response.stop_details.category if response.stop_details else None
return {"declined": True, "kind": "policy", "category": category}
# 2. Soft decline: the model answered normally, in prose, with a "no".
# stop_reason is "end_turn" here, so only the text reveals it.
# Use the judge, not a regex on "I'm sorry" - phrasing varies too much.
verdict = judge_declined(text, original_request)
return {"declined": verdict.declined, "kind": "soft"}
Teams that only regex for apology strings undercount soft declines badly: a model that answers a different, safer question has refused without ever apologizing. Prompt-injection defenses are a related but separate axis, covered in the prompt injection guide.
How to Run LLM Evals in CI
Wrap the runner in pytest, gate on the baseline rather than an absolute number, and only trigger the job on paths that can change behavior. A full eval on every commit to the README will get the whole suite disabled within a month.
import json, pytest
from evals.run_eval import run, load_cases
from evals.compare_runs import compare, wilson
BASELINE = json.load(open("evals/baselines/2026-07-14.json"))
CRITICAL = {"faq-014", "order-003", "refuse-011"}
@pytest.fixture(scope="session")
def results():
return run(load_cases("evals/cases"), open("prompts/support_agent.v7.md").read(),
model="claude-opus-5", scorers=SCORERS)
def test_no_critical_regression(results):
broken = compare(BASELINE, results)["broken"]
assert not (CRITICAL & set(broken)), f"critical cases broke: {broken}"
def test_pass_rate_holds(results):
k = sum(r["passed"] for r in results); n = len(results)
lo, _ = wilson(k, n)
base = sum(r["passed"] for r in BASELINE) / len(BASELINE)
assert lo >= base - 0.03, f"{k}/{n}, lower bound {lo:.3f} vs base {base:.3f}"
def test_refusal_balance(results):
assert rate(results, "must_refuse") >= 0.95
assert rate(results, "should_answer") >= 0.90
Gating on the interval's lower bound rather than the point estimate is what keeps the suite from crying wolf. The critical-case test earns its keep too: a named list that must never break, checked individually, immune to being averaged away.
name: evals
on:
pull_request:
paths: ['prompts/**', 'src/**', 'evals/**']
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: '3.12'}
- run: pip install -r requirements-dev.txt
- uses: actions/cache@v4
with:
path: .eval_cache
key: eval-${{ hashFiles('prompts/**', 'src/**') }}
- run: pytest tests/test_eval_gate.py -v --junitxml=report.xml
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- uses: actions/upload-artifact@v4
if: always()
with: {name: eval-results, path: 'evals/out/**'}
Upload the artifacts even when the job fails, because the whole value of a failed eval is being able to read the outputs that failed. If your pipeline conventions need shoring up, the CI/CD pipeline guide covers the surrounding machinery.
Why Offline Eval Diverges From Production
Your offline number will be higher than reality, usually by a lot. This is expected. The useful move is to measure the gap continuously rather than argue about which number is correct.
Input drift
The set was assembled at launch. Three months on, users ask about features that did not exist then, in phrasings nobody anticipated.
Pinned context
Offline runs pin retrieved passages for reproducibility. Production retrieves live from an index that has been reindexed, expanded, and partly corrupted since.
Single turn
Cases are one exchange. Real sessions run eight turns with accumulated context and a user who changed their mind in turn four.
No adverse conditions
Offline runs never hit a rate limit, a truncated output, a tool timeout, or a malformed upstream response. Production hits all four before lunch.
A fifth cause is self-inflicted: every case exists because you already fixed the bug it represents, so the set skews toward solved problems, and any prompt you tuned until the eval went green is a prompt fitted to the eval. Both push the offline number up without helping users. The holdout slice exists to bound this.
The fix is online eval. Log every production request with its inputs, retrieved context, output, latency, and tokens. Each day, sample 1 to 5 percent of that traffic and run the same scorers over it. You now have two series: offline says whether a change helped, online says whether users got what offline promised. Divergence is itself the signal that the set has gone stale.
Eval Tools Compared, Honestly
Every tool below solves the runner, and none of them solves the eval set. Pick based on where your team already lives rather than on a feature matrix, because the migration cost between them is a day and the cost of choosing badly is mostly boredom.
| Option | What it is | Fits when | Tradeoff |
|---|---|---|---|
| pytest plus your own scorers | What this guide builds. A few hundred lines you own. | Scorers are mostly deterministic and domain-specific | You build caching, reporting, and the diff view |
| promptfoo | Open-source CLI. Cases and assertions in YAML, side-by-side matrix output. | Comparing prompts or providers, and you want a fast local loop | YAML gets unwieldy once scoring needs real logic |
| Ragas | Open-source RAG metrics: faithfulness, answer relevancy, context precision and recall. | RAG, when you want standard metric definitions | Judge-driven, so calibrate before trusting them |
| DeepEval | Open-source, pytest-flavored assertions over LLM outputs. | You want eval to look and feel like the tests you already write | Opinionated metric set you may end up overriding |
| LangSmith | Hosted tracing plus datasets and eval runs, from the LangChain team. | Already tracing with it, or built on LangChain | Hosted; your data leaves your infrastructure |
| Braintrust | Hosted eval and observability with strong run-to-run diffing. | Multiple engineers changing prompts and needing a shared history | Commercial and hosted |
| Arize Phoenix | Open-source observability with tracing and eval, self-hostable. | You want a UI over traces without sending data outward | More observability than eval runner; you still write scorers |
| Inspect AI | Open-source framework from the UK AI Security Institute. | Safety and capability evals needing defensible methodology | Heavier than most product teams need |
Whatever you pick, keep the cases in your own repo in a plain format. Tools change; a JSONL file of labeled cases is the asset, and it should never be locked inside a vendor's database.
When Not to Build an Eval Suite
Three cases where the honest answer is to skip it. If you are prototyping and the definition changes weekly, a set built now measures something that will not exist in a fortnight; read outputs by hand and keep notes. If your application is a classifier with a fixed label set, you need a confusion matrix, not judges and rubrics. And if you have no users yet, resist inventing 200 synthetic cases: 20 real ones from five people beat them every time.
One caution. Fine-tuning changes the calculus, because a fine-tuned model can memorize eval cases that leaked into training data. Keep the set out of any training corpus and check for overlap before believing a large jump; the fine-tuning guide covers the data hygiene.
Frequently Asked Questions
How many examples do I need in an LLM eval set?
Enough that the confidence interval is narrower than the change you care about. A pass rate of 27 out of 30 reads as 90 percent, but the 95 percent Wilson interval runs from 74.4 to 96.5 percent, a spread of 22 points. At 180 out of 200 the same 90 percent gives 85.1 to 93.4 percent, a spread of 8 points. A five-point regression is invisible at 30 cases. Start at 30 to 50 per capability to catch outright breakage, then grow toward 150 to 300 before any number blocks a deploy.
Is LLM-as-judge reliable enough to gate a release?
It is reliable for what it can see and unreliable for what it cannot. A judge does well at checking whether an answer is grounded in a supplied passage, follows a format, or stayed on task, and at picking which of two answers a reader would prefer. It is weak on domain facts it was never given. Judges also carry position bias, a preference for longer answers, and a tilt toward their own model family. Hand-label 50 to 100 outputs, run the judge on the same set, and compute Cohen's kappa. Above roughly 0.6 it is usable as a gate; below that, fix the rubric first.
How do I measure hallucination in a RAG system?
Do not measure truth, measure groundedness. Split each answer into atomic claims and ask whether the retrieved context supports each one, then report the share of answers with at least one unsupported claim. That is checkable because the context is right there, whereas world truth is not. Underneath it, add a deterministic layer: a cited document ID absent from the retrieved set is an automatic failure needing no judge call. Track groundedness separately from retrieval recall, since an answer grounded in the wrong documents is still wrong.
Why do offline eval scores not match production quality?
Five causes dominate. The set was frozen at launch while user inputs drifted. Offline runs pin retrieved context while production retrieves live from a changing index. Evals are single-turn while production accumulates history. Offline runs never hit timeouts, truncation, or tool failures. And the set is selection-biased, because engineers add cases after fixing them. The fix is online eval: sample production traffic daily, score it with the same scorers, and track the gap between the two numbers as its own metric.
Method notes: all confidence intervals in this guide are Wilson score intervals at 95 percent; the paired comparison uses an exact two-sided McNemar test; the judge example uses the Anthropic Python SDK with structured outputs.