Evaluating AI Systems: Measure, Don't Guess

Everyone starts by eyeballing outputs. That works for the first demo and fails the moment you change anything. You tweak a prompt, swap a model, add a tool — and you have no idea whether you made things better, worse, or just different. Vibes don't compare, don't scale, and don't survive a teammate.

An eval replaces the gut-check with a repeatable measurement: run a fixed set of inputs through your system, score each result the same way every time, and report an aggregate number. Now “better” means the number went up, and you can prove it. Evals let you catch regressions, compare candidates fairly, and set a bar a release has to clear.

Vibes still have a place — for a brand-new feature with no data, a five-minute manual pass tells you if you're in the right ballpark. But the instant a system is worth shipping, it's worth an eval. The goal isn't a perfect score; it's a trustworthy, moving score.

Your eval is only as good as its dataset. A golden set is a curated collection of inputs paired with what a correct response looks like — either an expected output or a rubric describing one. Build it from three sources:

• Real usage — sample actual queries, not ones you imagined. This anchors the eval to reality.
• Hard cases — the ambiguous, adversarial, and multi-step queries where the system tends to break.
• Edge cases — empty inputs, wrong language, malformed data, out-of-scope asks.

Fifty well-chosen examples beat five thousand random ones. Version and freeze it — a golden set that quietly changes makes every historical score meaningless. And keep it out of your prompts and few-shot examples; a test the model has already seen isn't a test.

{"id": "refund-ambiguous-01",
 "input": "I want my money back for the thing I bought",
 "expected": {"intent": "refund_request", "needs_clarification": true},
 "tags": ["intent", "ambiguous"]}

Two questions, two kinds of metric. Task-success asks: did it get the right answer? That's exact-match for structured outputs, semantic similarity for free text, or a rubric pass/fail for open-ended work. It's the outcome the user cares about.

Trajectory asks: did it get there sensibly? For agents, the path matters as much as the destination — did it call the right tools, in a reasonable order, without looping? Trajectory metrics include tool-call correctness, number of steps, and cost or latency per task.

You need both. An agent that reaches the right answer after twelve redundant tool calls is expensive and fragile even when task-success says “pass.” Conversely, a clean-looking trajectory that ends on the wrong answer is still a failure. Track outcome as your headline number and trajectory as the diagnostic that tells you why a run passed or failed.

For open-ended outputs where exact-match is useless, you can use a strong model to grade against a rubric. This is LLM-as-judge, and it scales human judgment cheaply. It also inherits some ugly biases you must design around:

• Position bias — in pairwise comparisons, judges favor whichever answer came first. Randomize the order and average.
• Verbosity bias — judges reward longer answers. Tell the rubric to grade correctness, not length.
• Self-preference — a model tends to prefer text from its own family. Cross-check with a different judge model.
• Non-determinism — the same output can score differently across runs. Use a clear rubric and consider multiple samples.

The essential safeguard: calibrate the judge against human labels. Hand-grade a slice, compare the judge's scores, and only trust the judge on the rest once the two agree. A judge you haven't calibrated is a vibe with extra steps.

JUDGE = """Score the RESPONSE against the RUBRIC from 1-5.
Grade ONLY factual correctness and completeness. Ignore length and style.
Return JSON: {"score": int, "reasoning": str}.

RUBRIC: {rubric}
QUESTION: {question}
RESPONSE: {response}"""

An eval you run once is a report. An eval you run on every change is a safety net. Wire your golden set into CI so any prompt edit, model swap, or dependency bump reruns the suite and gates the merge on a pass-rate threshold. Now a well-meaning prompt tweak that quietly breaks the refund flow gets caught in the pull request, not in production.

A healthy tooling stack usually pairs two things: a lightweight framework that runs in CI, and a platform for tracking results and human annotation over time. Verified options:

• promptfoo and DeepEval — developer-first, CI-native eval frameworks.
• Ragas — metrics built specifically for RAG pipelines.
• Inspect AI — an open framework from the UK AI Safety Institute.
• OpenAI Evals — a registry-style, open-source harness.
• LangSmith and Braintrust — managed platforms for tracing, dashboards, and human review.

You don't need all of them. One CI framework plus one dashboard is the common, sufficient setup.

Every eval framework is the same four-step loop underneath. You can build it in an afternoon in any language; reach for a framework once you outgrow the basics.

results = []
for case in load_golden_set("evals.jsonl"):
    output  = my_system(case["input"])            # 1. run
    passed  = score(output, case["expected"])     # 2. score each case
    results.append({"id": case["id"], "pass": passed, "output": output})

rate = sum(r["pass"] for r in results) / len(results)   # 3. aggregate
report(results, rate)                                    # 4. report + diff vs baseline

assert rate >= 0.90, f"Regression: pass rate {rate:.2%} below 0.90 gate"

The score() function is where the design lives: exact-match for structured fields, string/semantic similarity for text, an LLM-judge for open-ended answers, or a mix keyed on the case's tags. Log every failing case's full input and output — the aggregate tells you whether you regressed; the failures tell you where.

A number can lie if you don't handle it carefully.

• Mind the sample size. A 4% jump on 25 examples is noise. Report how many cases the number is built on, and don't over-read tiny deltas.
• Slice, don't just average. An overall 90% can hide a 40% pass rate on your hardest category. Break results down by tag.
• Track over time. One score is a snapshot; a trend line is the real signal. Store every run.
• Don't overfit the eval. If you tune endlessly against the same set, you optimize for the test, not reality. Keep a held-out set you rarely look at.

The healthiest teams treat evals as a living instrument — the golden set grows as new failures surface in production, and the bar rises as the system improves. The number is a servant, not a trophy.