Eval-Driven Development

Concept of the skill

Eval-driven development is the practice of building LLM-integrated systems by writing evaluations before and alongside the system, with each eval defining a behavioral criterion the system must satisfy on a representative input set, and the suite's aggregated signal gating every change to prompt, model, retrieval, context assembly, scaffolding, tooling, or orchestration. Five primitives: (1) eval dataset — curated input examples representing production plus edge cases, prior failures, and held-out slices, typically JSONL of {input, reference} records checked into version control; (2) evaluation function — per-example grader producing a score, with mechanism chosen from programmatic, model-graded, human-graded, preference comparison, trace check, or hybrid; (3) aggregation — statistical summary across the dataset (pass-rate, weighted pass-rate, stratified slice scores, distribution) with sample size and either a confidence interval or a defined minimum-detectable-change threshold, plus cost/latency side metrics; (4) iteration loop — eval → diagnose → change one meaningful variable → re-eval → compare, with the suite stable while the system changes; (5) regression budget — per-eval/per-slice merge policy: gating (no regression allowed), optimizing (improvement gates merge), watchful (tracked, not gated), human-signoff, or cost-aware threshold (a pass-rate gain only counts if it clears the eval's cost/latency budget — a 1pp quality gain at 3× inference cost is a regression).

Replaces "ship LLM systems based on how the model feels at the desk" (and benchmark shopping) with empirical measurement that distinguishes "the new prompt works better" from "the new prompt works better on the five examples I happened to try." Solves the problem that LLM system behavior is stochastic (every run is a sample, not a deterministic pass/fail), the input space is open-ended (the developer's pet examples are not a representative sample of what users will throw at it), model behavior changes across releases (a feature that failed last quarter may now be viable), and traditional binary unit tests do not capture the distributional nature of LLM outputs. The eval suite is the empirical measurement instrument that makes shipping decisions defensible and changes gateable: each change has a baseline, a representative private eval set, a comparison method, and a documented decision policy. The discipline's hard part is not writing evals — it is choosing what to measure, encoding the choice into a grader the team agrees with, sampling a dataset that represents production, distinguishing a real improvement from sampling noise, tracking high-risk slices separately from the average, calibrating model-graders against humans, feeding sanitized production failures back into the next eval version, and resisting Goodhart's Law as the eval suite becomes the optimization target. Teams that get this right ship systems whose quality matches their stated definition of "good"; teams that get this wrong ship systems that ace evals and disappoint users.

Distinct from testing-strategy, which owns deterministic-software testing where every run is binary pass/fail and the main decision is the cheapest honest test level — this skill owns LLM evaluation where every run is a sample from a distribution and pass-rate (or a score distribution) is the unit of judgment; the disciplines share vocabulary (suite, gate, regression) but the math underneath differs. The exact construction of eval task sets, rubrics, graders, hard negatives, trace schemas, and thresholds is separate from this skill; this skill owns the development discipline that uses those artifacts to gate whether prompt/model/retrieval/tool/orchestration changes ship. Distinct from evaluation, which owns general scoring frameworks and result interpretation — this skill owns the repeated LLM-system change gate. Distinct from tool-call-flow, which owns the protocol cycle by which a model invokes tools — this skill owns the discipline of measuring whether that cycle produces correct behavior; tool-call and trajectory evals are a specialization of the general pattern. Distinct from prompt-injection-defense, which owns the security property and threat model — this skill owns the measurement of whether the property holds (red-team evals against an injection corpus are one application). Distinct from error-tracking and observability-modeling, which own runtime measurement of deployed systems — this skill owns offline pre-deployment measurement while using production signals as inputs for dataset refresh; one is not used as a substitute for the other. Eval-driven development is to LLM system engineering what crash-test ratings are to automotive safety — you do not ship a car based on how well it parked in your driveway; you ship it after a battery of standardized tests on representative crash scenarios, with the pass-rate against named criteria as the gating signal. A score of 4.3 stars across the suite is the only defensible claim of 'safer'; a developer's intuition that 'the new model feels smarter' is the unmeasured equivalent of 'I drove it home, it seemed fine.' And just as a crash lab that the manufacturer trains its cars to detect stops measuring real safety, an eval the model can recognize as a test stops measuring real behavior. The wrong mental model is that LLM evals are "unit tests for prompts" — binary pass/fail, written once, checked in CI, ship on green. They are not. The fundamental property is distributional and lifecycle-based: every run is a sample from a stochastic system, and the unit of judgment is pass-rate over a sampled population with a sample size and confidence interval. A test that runs once and "passes" tells you nothing about the distribution; pass-rate at n=20 has huge uncertainty; pass-rate at n=500 starts to be informative. Adjacent misconceptions: that a higher headline pass-rate is a real improvement (it is not until the delta's confidence interval excludes zero — a paired bootstrap on a shared dataset is the cheap default, McNemar's test is the right check for paired pass/fail outcomes, and halving the effect you want to detect costs four times the samples); that a single headline number is sufficient (it is not — it can hide a high-risk slice regressing while the average improves; a panel of independent measures and slices is harder to over-fit, and HELM's multi-metric framing exists precisely as a counter to single-metric Goodharting); that public benchmarks (MMLU, HumanEval, BIG-bench, GAIA, MT-Bench, τ-bench) gate system-specific shipping decisions (they do not — benchmarks predict how a model will do on the exact tasks they contain, not on your system's user inputs; use them for model-selection grounding, and treat a sudden benchmark jump as a contamination hypothesis, not an automatic capability win); that higher eval scores are always improvements (they are not — Goodhart's Law: when the eval becomes the optimization target, it ceases to be a good measure; a 100% pass-rate is a measurement failure on a capability/gating suite, not a victory, because a suite the system aces has stopped discriminating — though a regression suite near 100% is doing its job, see the carve-out in § The 100% Pass-Rate Trap); that model-graded evals are reliable without calibration (they are not — model-graders have verbosity bias, position bias in pairwise comparisons, self-preference bias toward their own family's outputs, and correlated error with the system being graded; the fix is mechanical — judge from a different top-tier family, swap positions, ensemble across families — not a "be objective" instruction in the grader prompt, because the bias lives in the embedding distribution, not the surface style); that you can trust a metric without reading the transcripts (you cannot — error analysis on the actual traces is how you learn whether the grader itself is working and what the failures really are); that final-output grading is sufficient for multi-step agents (it is not — an agent that reaches the right answer through a broken or unsafe trajectory will pass a final-output eval and fail in production; multi-step systems need trajectory-aware evals that score planning, tool selection, and execution); that the model under test is fixed (it is not — every model upgrade or vendor switch changes the system under test and must be re-gated against the private suite, including high-risk slices, cost, and latency); and that offline evals replace production telemetry (they do not — production telemetry measures the actual user-facing system under actual load, and every production incident is a candidate eval case after sanitization; the two are complementary, not substitutable). Public benchmarks, vendor dashboards, and LLM judges are useful inputs, not shipping authority by themselves.

Coverage

The practice of building language-model-integrated systems by writing evaluations before and alongside the system, then using the eval suite's measured signal to gate every meaningful change. Covers:

• The statistical (not binary) nature of LLM evaluation: pass-rate, score distribution, repeated runs, slice variance, confidence intervals, and minimum detectable change.
• The five primitives: dataset, evaluation function, aggregation, iteration loop, regression budget.
• Where eval datasets come from (production logs, user-feedback capture, hand-authoring, prior failures, adversarial generation, synthetic generation) and how origin determines representativeness.
• How a suite starts small and grows — a handful of real-failure cases is a legitimate beginning; statistical power is earned by accretion, not required on day one.
• The bootstrap-vs-certification distinction: a 20–50 case early suite can drive coarse development feedback, but it cannot certify small deltas unless its MDE supports the claim; every suite is labeled with the posture (bootstrap, regression, capability, launch gate) that says which claim it warrants.
• Repeated-trial metrics for stochastic agents: pass@k ("at least one success in k attempts") versus pass^k ("all k attempts succeed"), and why the product promise — one good candidate vs reliable-every-time behavior — decides which one gates.
• Trace-first debugging versus repeatable eval runs: traces diagnose workflow failures; versioned datasets and eval runs gate comparable changes.
• Benchmark retirement and deprecation hygiene: public benchmarks, shared evals, and private suites must be marked stale, contaminated, saturated, or retired when they stop measuring the intended capability.
• The judgment-mechanism taxonomy (programmatic / model-graded / human-graded / preference comparison / trace-hybrid) and its cost dimension.
• Error analysis — reading transcripts/traces before trusting any metric — as the diagnostic engine of the loop.
• The eval surface stack: final response, trajectory/tool-use, retrieval/context use, safety/boundary, side effects, cost/latency — and which layer must block merge.
• RAG-specific eval surfaces: context precision, context recall, faithfulness/groundedness, answer relevance, citation support.
• The discipline of distinguishing a real improvement from sampling noise: paired difference tests, McNemar's test for paired binary outcomes, bootstrap confidence intervals on the delta, minimum-detectable-effect, clustered standard errors, and high-risk slice vetoes.
• The distinction between final-output and trajectory evaluation for multi-step agents.
• Continuous evaluation and model/vendor-upgrade gates: rerunning private evals before adopting a new model, API feature, tool runtime, or judge model.
• The distinction between system-specific evals and canonical public benchmarks (MMLU, HumanEval, BIG-bench, GAIA, MT-Bench, HELM, SimpleQA, SWE-bench, τ-bench).
• Why higher scores are not always improvements (Goodhart's Law, including suite saturation at 100%), and the eval-awareness/contamination risk that erodes benchmark validity — including the structural worsening of eval-awareness with model capability.
• The pre-registration / spec-first discipline that keeps the iteration loop from degrading into post-hoc p-hacking.
• Cost-aware regression budgets: when a quality gain justifies higher latency, token spend, tool calls, or human-review load.
• Offline evals vs production telemetry: the graded offline→shadow→canary→A/B→full rollout spectrum, and how live traces, feedback, incidents, and A/B tests feed the next eval version without replacing pre-deployment gates.
• The eval-lifecycle archetypes (acceptance, regression, calibration, red-team, cross-model, drift-refresh).

Philosophy of the skill

Building LLM-integrated systems without evals is shipping airplanes based on how good the model feels at the desk. The system's behavior is stochastic, the input space is open-ended, the model may change under you, and the developer's pet examples are not a representative sample of what users will throw at it. An eval suite is the empirical measurement instrument that lets a team distinguish "the new system is better" from "the new system is better on the five examples I happened to inspect."

The discipline's hard part is not writing evals. It is choosing what to measure, encoding the choice into a grader the team agrees with, sampling a dataset that represents production, distinguishing a real improvement from sampling noise, calibrating subjective graders, holding the suite stable while the system changes, and resisting the gravitational pull of Goodhart's Law as the eval suite becomes the optimization target. Teams that get this right ship systems whose quality matches their team's stated definition of "good." Teams that get this wrong ship systems that ace evals and disappoint users.

Eval-driven development is not test-driven development with extra noise. It is empirical engineering applied to systems whose behavior is a distribution rather than a value. The vocabulary overlaps with testing — suite, gate, regression, CI — but the unit of judgment differs. A deterministic unit test asks "did this exact behavior happen?" An LLM eval asks "how often, under which slices, at what cost, with what uncertainty, and against whose definition of quality?"

Start small; do not wait for a perfect suite. The barrier to entry is low and the cost of delay is high: a starter set of 20–50 tasks drawn from real failures is a legitimate beginning, not a placeholder (Anthropic's agent-eval guidance). Early in development the effect sizes of changes are large, so even a small suite separates good from bad; the statistical machinery below matters most later, when the easy wins are gone and you are arguing over single percentage points. The failure mode is not "started with too few cases" — it is "never grew the suite, and never read the transcripts."

The Five Primitives In Practice

Three invariants keep the loop honest:

• Keep the eval suite stable while the system changes. If the dataset, grader, and system all change at once, the comparison anchor is gone and the team is doing parallel experiments.
• When the eval itself changes, version it as a new measurement instrument. Re-baseline the old and new system on the new eval if you need continuity; a delta is only comparable across runs on the same dataset version.
• Pre-register the comparison before seeing candidate results. At minimum, write the hypothesis, dataset version, primary metric, slice vetoes, judge version, sample size or repeat policy, cost/latency budget, and merge rule. This is the structural defense against post-hoc rationalization (see § Iteration Loop Discipline).

Where Eval Datasets Come From

The first primitive — the dataset — is only as good as its representativeness, and origin determines what population the result can speak for and which blind spots the suite will have. Six common origins, ordered roughly from closest-to-deployment to furthest:

A healthy suite blends origins and labels each case with provenance: production-sampled inputs anchor representativeness, hand-authored + adversarial cases pin the spec and the boundaries, prior-failure cases prevent regressions, and synthetic generation fills volume — with the mix and each stratum's weight chosen deliberately, not by whichever data was easiest to collect.

Bootstrap Suites: Start Small, But Label The Claim

The "start small" point from § Philosophy has a discipline attached: the claim a suite can support must match its sample. A starter set of 20–50 well-chosen tasks from real failures, manual smoke checks, support issues, and product requirements is enough to expose coarse regressions and make the team's definition of success concrete — it is a development steering wheel, not a certification instrument. A bootstrap suite can justify "this change fixed the obvious failure class" or "the new model is worth deeper evaluation"; it cannot justify "a 2pp lift is real" unless the design's minimum detectable effect supports that claim (see § The Statistics of "Is This Better?"). Effects worth shipping shrink as the system improves, so mature systems need larger, harder, stratified suites, repeated trials, or paired comparisons.

The operational rule is to label every suite with its use posture — this is the epistemic-warrant companion to the cadence/gating view in § Eval Lifecycle Archetypes:

What To Evaluate — The Eval Surface Stack

"Run the evals" is underspecified until you name which surface each eval scores. For simple LLM features, final-output quality may be enough; for agents it is usually too shallow. A complete LLM system has several distinct evaluable surfaces, and grading only the final response — the most common mistake — leaves the others unmeasured.

A suite that covers only the first row reports a single number that hides failures on every other surface. Decide which surfaces your system actually has, and put at least one eval on each that matters. This skill decides how those surfaces become a change gate; constructing the exact rubrics and trace schemas is a task-specific artifact concern.

RAG Eval Surfaces

Retrieval-augmented systems have a distinct, well-named set of eval dimensions because a RAG answer can fail in two independent places — the retriever fetched the wrong passages, or the generator ignored/contradicted the right ones. A green final answer can hide an unreliable retriever, and a faithful answer can still be useless if the retriever missed the necessary source. Naming the layers separately localizes the failure instead of blaming "the model" (RAGAS framing):

Report the failure location: retrieval miss, ranking/noise problem, context-packing loss, synthesis hallucination, bad refusal, or citation mismatch. The regression budget should say which layer blocks merge and which only triggers diagnosis. These are eval surfaces to name and measure, not a RAG-metric construction manual — building the retrieval corpus, the rubric, and the graders is the task-specific work of building the retrieval corpus, rubric, and graders.

Safety And Red-Team Eval Gates

Safety evals need named gates, not just a generic "safety score." Separate attack success from false-positive refusal and severity.

This skill owns making those signals part of the change gate. The threat model, attack taxonomy, and defense mechanism belong to prompt-injection-defense or the relevant safety/security skill.

Judgment Mechanism Selection

A practical default: programmatic checks for invariants, model judges for semantic scale, preference comparison for variants, and human review for calibration and high-risk decisions.

Cost is a design axis, not an afterthought. Per-grade cost multiplied by suite size multiplied by run frequency is a real budget, and it shapes which mechanism goes where. A 500-example suite graded by a model at ~$0.05/grade is ~$25 per run — runnable on every PR; the same suite under human grading at ~$5/grade is ~$2,500 per run — a quarterly-calibration activity, not a CI gate. The design rule that falls out: put the highest-frequency runs (per-commit gates) on the cheapest adequate mechanism (programmatic, then model-graded), and reserve human grading for the low-frequency calibration and ambiguous-case sampling that only humans can do. Cost also belongs in the gating decision itself — see the "cost-aware threshold" regression-budget policy and § Cost-Aware Evaluation.

Read The Transcripts — Error Analysis Is The Engine

A metric is a summary, and a summary you have not traced back to specific transcripts is a number you do not yet understand. Before trusting any aggregate — and especially before trusting a model-graded aggregate — read a sample of the actual traces: the inputs, the full output, the tool calls and intermediate state, and the grade the grader assigned. This is error analysis, and it is the diagnostic engine of the iteration loop, not an optional polish step. "You won't know if your graders are working well unless you read the transcripts and grades from many trials" (Anthropic's agent-eval guidance).

Error analysis answers questions a pass-rate cannot:

• Is the grader actually right? A model-graded eval can report 85% while silently mis-grading a recurring failure shape; only reading graded transcripts catches a broken grader before it pollutes every downstream decision.
• *What are the failures really?* Reading a dozen failing traces usually clusters into 2–3 named failure modes (wrong tool, dropped context, over-refusal, hallucinated citation) far faster than staring at the scalar score — and those named modes become the next eval cases and the next fix.
• Is a "pass" a real pass? A trace that reached the right answer through a broken or unsafe path passes a final-output grader; only the trace reveals it.

The loop is: read traces → name failure modes → encode them as cases/graders → then optimize the metric. Skipping straight to "the number went up" is how teams optimize a metric they have never grounded in reality.

The Statistics of "Is This Better?"

A pass-rate is an estimate, not a measurement. "Version B scored 3 points higher" is not a shipping argument until you have asked whether the gap survives sampling noise. This is the single most common place eval-driven development degrades into eval-flavored vibes: a team reads two numbers, declares a winner, and ships a change the data does not actually support. (The statistics matter most once the suite is past its starter phase — early, large-effect changes are visible even at n=20; single-percentage-point arguments are where the machinery below earns its keep.)

Report sample size with every pass-rate, mean score, or win rate. A bare number with no n is not interpretable.

Compare the delta, not two separate intervals. The decision rule is whether the *confidence interval on the difference (B − A) excludes zero* (or clears the predeclared regression budget) — not whether A's and B's individual intervals fail to overlap (the overlap check is needlessly conservative and throws away power).

Prefer a paired test on a shared dataset. Run both system versions on the same eval examples and analyze the per-example score differences. Because scores on the same question are positively correlated across versions, the paired estimator removes that shared variance "for free" — a paired t-test or Wilcoxon signed-rank produces a substantially tighter confidence interval on the delta than the unpaired equivalent, at the cost of roughly 2× inference (you run both arms). How much tighter is not a fixed quantity: it scales with the within-example correlation between the two arms, so the gain is large when versions agree on most examples and modest when they do not — there is no universal "orders of magnitude" multiplier to quote. When you cannot pair (different datasets), fall back to the unpaired interval and expect to need far more data.

For paired binary pass/fail outcomes, name the test. McNemar's test (or an exact binomial test on the discordant pairs) is the usual check for whether B fixes more cases than it breaks. The interesting count is not total wins; it is A fail / B pass versus A pass / B fail.

Bootstrap the delta when assumptions are shaky. For small, skewed, or non-normal datasets, compute delta_i = score_B_i − score_A_i for each shared example, resample the delta vector with replacement (e.g. 1,000–10,000 resamples), recompute the mean/pass-rate delta, and use the 2.5th and 97.5th percentiles as the approximate 95% interval. Ship only when that CI sits entirely on one side of zero. When central-limit assumptions hold and n is a few hundred or more, a closed-form paired interval (Δ ± 1.96 × SE(difference)) is sufficient and bootstrapping is optional.

Know your minimum detectable effect (MDE) before you run. Sample size scales quadratically with the inverse of the effect you want to detect: detecting a gap half as large requires 4× the samples. There is no universal "N examples per 1% gap" constant — the requirement depends on the design and the pass-rate. For an unpaired one-sample 95% CI with ±1pp half-width, the normal approximation needs roughly 1.96² · p(1−p) / 0.01² examples: ≈9,600 at p=0.5, ≈6,100 at p=0.8, ≈3,500 at p=0.9 — thousands, not hundreds, to pin a single percentage point. As a rough feel for smaller suites, a binary pass-rate at n=100 has a 95% half-width of about ±8pp near 80% pass-rate and about ±10pp near 50%; n=400 cuts those roughly in half. *These figures are the unpaired case — and they are why teams that actually ship on small eval suites run paired comparisons, which need far fewer examples for the same MDE (how many fewer depends entirely on the arm correlation, so no fixed per-1pp number is defensible for either design). The practical consequence: a small eval resolves only coarse effects — treat any delta below your design's MDE as "no signal," not "small improvement," and compute that MDE for your* design and pass-rate rather than trusting a rule of thumb. For finer MDE you need more examples, repeated runs, or token-probability scoring.

Account for correlated examples (clustered standard errors). If your dataset contains groups of related examples — multiple questions from one document, paraphrases, translations, repeated tasks from one user — the effective sample size is the number of clusters, not the number of rows. Naively treating correlated rows as independent can understate the true standard error by ~3×, turning noise into a false "significant" result. Cluster the standard error on the grouping variable, or treat the cluster count as the effective n.

Shrink within-question noise with repeats. A stochastic system gives a different score on the same input across runs. Generating K samples per question and averaging reduces the within-question variance by a factor of K; when token-probability scores are available, that within-question noise can be eliminated entirely. Repeat runs whenever the system, tools, environment, or judge is stochastic enough that one run can flip the result.

Choose the repeated-trial metric that matches the product promise (`pass@k` vs `pass^k`). For stochastic agents, repeated runs can answer two opposite questions. pass@k asks whether at least one of k attempts succeeds; it rises as k increases and is appropriate when the product can cheaply try multiple candidates and keep the first valid one. pass^k asks whether all k attempts succeed; it falls as k increases and is the right consistency metric when a user expects the agent to work every time. A change can improve pass@10 while making pass^3 worse — it found more lucky successes but became less reliable. Record which repeated-trial metric gates the change before running the eval, because the two reward opposite behaviors.

Report slices separately, and treat high-risk slices as veto gates. Break results out by task type, difficulty, user segment, language, tool path, safety class, context source, model family, and known prior failures. A +5-point average does not justify a critical privacy, safety, tool-side-effect, or grounding regression on a slice.

Reporting standard. A defensible eval result reports: mean score, standard error (CLT-based or clustered), the 95% CI on the delta, sample size n, cluster count, the slice table, the MDE or threshold, the repeat-run policy, the cost/latency delta, and — for a paired comparison — the correlation between the paired arms. A bare "B scored higher" with none of these is an unverified claim.

Boundary note: constructing the rubric, hard negatives, and acceptance thresholds is the task-specific work of constructing the rubric, hard negatives, and acceptance thresholds; interpreting whether the delta is real and gating the change on it is this skill's. The statistics live here because they are the gating decision.

Iteration Loop Discipline

The eval-driven iteration loop is the development cycle. Run the suite, diagnose failures and slices, choose one meaningful change, re-run the same suite, gate the change on the regression budget.

baseline eval
  -> diagnose failures and slices (READ THE TRANSCRIPTS, not just the score)
  -> choose one meaningful change
  -> re-run the same eval suite
  -> compare against regression budget
  -> merge, iterate, or revert

The "diagnose" step is error analysis, not metric-watching (see § Read The Transcripts). Allowed system changes include prompt wording, system/developer instructions, retrieval query strategy, context packing, tool schema, tool-call policy, model choice, reasoning effort, output format, guardrail policy, and orchestration. The suite should remain stable during the comparison. Change one thing per iteration so the eval delta is attributable; if the budget allows multiple changes, gate them as a batch but expect harder root-causing when the batch regresses.

Pre-register the decision rule before you run (spec-first, anti-p-hacking). The structural defense against fooling yourself is to write down what you will measure, the threshold, and the ship/no-ship rule before you see the result — not after. Before a candidate run, write the eval spec as if it were a test plan: baseline system, candidate change, dataset version, primary metric, secondary metrics, slice vetoes, judge/grader version, sample size or repeat count, MDE, cost/latency ceiling, and merge rule. After the run, record whether the predeclared rule passed. An iteration loop that decides the metric and cutoff after looking at the numbers is p-hacking with extra steps: with enough sliced views (this subset, that stratum, this rephrased grader) some comparison will always cross significance by chance, and a team that picks the favorable slice post-hoc ships noise. Treat any post-hoc slice that "looks better" as a *hypothesis