Chapter 3. Evaluation Design

Learning objectives

• Construct an evaluation set that represents the real-world usage distribution and separately addresses edge cases and rare high-risk cases.
• Compare the strengths and weaknesses of code-based grading, LLM-based grading, and human grading, and select the appropriate grading method for each criterion.
• Calibrate LLM graders against human grading, and determine whether an increase in score represents a genuine improvement.
• Distinguish the roles of capability evaluations and regression evaluations, as well as offline evaluations and online metrics, and design them together.
• Interpret evaluation results by category and decide between launch, conditional launch, or hold, along with supporting rationale.

Introduction

A team deployed a new prompt for a customer support bot and said, "We ran ten examples, and it has definitely improved." A week later, an issue was discovered where shipping fees were stated incorrectly only for international order inquiries. There was not a single international order among the ten examples. OpenAI calls this approach "vibe-based evals" and cites it as a classic failure pattern.

Evaluation is the only way to verify the quality of an AI feature with numbers, and those numbers serve as the basis for launch decisions. Therefore, evaluation design is not a job for developers alone. Product planners who defined what constitutes success (Chapter 2) must also take responsibility for determining which data and grading methods will verify that success. OpenAI recommends evaluation-driven development, running evaluations early and often, and states that evaluation is not a task with an end, but an ongoing process.

This chapter begins with how to build an evaluation set, examines the three grading methods and the order of selecting them, and explores how to make LLM graders reliable. Next, it distinguishes between capability evaluations and regression evaluations, as well as offline evaluations and online metrics, and finally covers how to interpret evaluation results to make launch decisions.

Key concepts

What is evaluation?

An evaluation (eval) is a structured test that measures the performance of a model or feature. OpenAI's definition is the same. An evaluation consists of three parts: an evaluation set that corresponds to exam questions, a grading method that scores the answers, and success criteria that correspond to the passing threshold.

By analogy, an evaluation is like a driver's license test. The test course (evaluation set) must resemble real roads, the examiner (grading method) must be consistent, and the passing threshold (success criteria) must be established in advance. If the test course consists only of a parking lot, one might pass yet still cause an accident on the road. If examiners award scores based on their mood, the same level of ability will be evaluated differently.

The five stages of the evaluation process

OpenAI describes the evaluation process in five stages:

1. Define evaluation goals. What are you trying to verify? This comes from the success criteria in Chapter 2.
2. Collect data. What data is needed to verify those goals?
3. Define metrics. How will you verify whether the success criteria have been met?
4. Run and compare. Run evaluations and compare between versions.
5. Evaluate continuously. Re-run evaluations whenever changes occur.

The output of the preceding stage is the input to the next stage. If you collect data without goals, you will not know what to collect; if you define metrics without data, you will not know what to apply them to. If the final stage is omitted, evaluation ends up as a one-time event before release, and subsequent changes are judged by feeling once again.

Representativeness of the evaluation set

The first evaluation design principle in Claude documentation is "Align with the task." This means that evaluations must reflect the distribution of actual tasks. OpenAI also cites an evaluation set that fails to faithfully reproduce production traffic patterns as the failure pattern of "biased design."

The easiest way to check representativeness is a comparison table:

When a table like this reveals significant discrepancies, the evaluation score cannot represent actual performance. Questions written off the top of a team member's head often consist of complete sentences in standard language, whereas actual users ask questions using abbreviations, typos, and mixed languages. The foundation of an evaluation set must be a randomly sampled set of real-world requests.

Edge cases and rare high-risk cases

A representative sample alone is not enough. Claude documentation emphasizes including edge cases in evaluations and provides four examples: irrelevant or non-existent inputs, excessively long inputs, in the case of chat, low-quality, harmful, or irrelevant user inputs, and ambiguous cases where human judgments diverge. OpenAI likewise advises including typical cases, edge cases, and adversarial cases together in evaluation data.

There is a temptation to exclude ambiguous cases on the grounds that "grading is inconsistent." However, those are precisely the cases that fail most frequently in real-world operations. Rather than removing them, tag them separately and review the results of that subset independently.

Rare but high-impact cases (urgent requests, safety-related questions, large-value transactions) appear only a handful of times in a random sample. A 0.4% category yields only around four instances in a sample of 1,000 cases, making it impossible to evaluate performance on that category. Such cases should be collected separately to form a dedicated subset of sufficient size, and passing status should be judged independently from the overall score.

Sources and size of the evaluation set

Evaluation data comes from multiple sources. OpenAI cites production logs, domain expert-curated data, synthetic data, and purchased data, and advises logging during development to mine good evaluation cases.

Regarding size, remember both "the more the better" and "start small." Claude documentation states that if automated grading is feasible, having a larger number of questions—even with slightly lower precision on individual gradings—is preferable to a small number of high-quality evaluations graded by hand by humans. Anthropic's agent evaluation article states that 20 to 50 simple tasks drawn from real-world failures make a good starting point. Starting small and expanding with failures discovered in production is the practical path.

Held-out sets and overfitting

If you revise a prompt dozens of times while looking at the same evaluation set, the prompt becomes tailored to that set, causing scores to appear higher than actual performance. This is just like a student who studies after seeing exam questions in advance; their score does not reflect genuine ability.

The Google PAIR guide explains that test sets must be kept as data the model has never seen to understand how well it actually works, and benchmark examples in Claude documentation also evaluate performance on a held-out test set. In practice, evaluation sets are divided into two parts: a development set reviewed during revisions, and a held-out set withheld during revisions and used solely for final determination. If development set scores increase significantly while held-out set scores remain flat, that improvement is very likely an artifact of overfitting to the set.

Three grading methods

Claude documentation divides grading methods into three categories:

Anthropic's agent evaluation article organizes this similarly. Code-based grading is fast, cheap, objective, and reproducible, but struggles with correct answers that differ from the ground truth only in expression; model grading can vary slightly from run to run, requiring calibration against human grading; and human grading serves as the gold standard, but is expensive and slow.

Order of selecting grading methods

The principle in Claude documentation is to "choose the fastest, most reliable, and most scalable method." For each criterion, ask the following questions in order:

1. Is there a defined ground truth that can be cross-checked? → Code-based grading.
2. Does it require judgment, but can be written as a rubric? → LLM grading (after verifying reliability against humans).
3. Is there no rubric yet, or is expert judgment required? → Humans grade a representative sample to establish baselines (rubrics and baseline data). If it involves specialized quality that is difficult to translate into a rubric, use an expert blind comparison itself as the pre-launch evaluation.

Even within a single feature, methods differ across criteria. For a refund guidance response, "Does the amount match the order system value?" is handled by code; "Is the policy interpretation correct?" and "Does the tone align with the guide?" are handled by an LLM grader equipped with a rubric; and the calibration of that grader and ambiguous cases are handled by humans. Attempting to grade every criterion with a single method sacrifices either cost or reliability.

Making LLM graders reliable

Claude documentation provides three tips for LLM grading: provide a detailed and unambiguous rubric, require specific outputs such as "correct/incorrect" or "1–5 points," and have it reason before producing a score. Example code in the same documentation also notes that evaluating with a model different from the one that generated the output is generally recommended.

[Rubric example: Policy interpretation in refund guidance responses]
Correct (correct): Stated refund eligibility and deadlines matching the order date, product condition, and payment method.
Incorrect (incorrect): Refund eligibility or deadlines differed from policy, or added conditions not found in policy.
If judgment is difficult, evaluate as "unsure".
Output: First write the rationale for judgment in one or two sentences, and on the last line write only one of correct / incorrect / unsure.

A rubric alone is not enough. OpenAI advises calibrating automated grading against human evaluation, and scaling up only when LLM graders are faster and cheaper while consistently aligning with human judgments. The procedure is as follows: humans grade a representative sample using the rubric; the same sample is graded with the LLM grader to observe agreement; criteria that diverge lead to rubric revisions; and once agreement exceeds the target, it is deployed across the entire dataset while being periodically cross-checked against human samples.

LLM graders have known biases. OpenAI cites position bias based on the presentation order of responses as a bias of LLM graders, and recommends pairwise comparison or pass/fail determinations as more reliable methods. When using pairwise comparison, grade twice with the order of the two responses swapped to reduce position bias. The tendency to rate longer responses more favorably regardless of content (length bias) is also common. Therefore, when scores jump significantly, first read the grading logs directly. Anthropic states that without reading through the logs and scores of many trials, one cannot know whether grading is functioning properly.

The role of human grading

Saying human grading should be avoided when possible does not mean eliminating human grading altogether. Human grading is indispensable in three areas: when initially creating rubrics and establishing baseline data, when calibrating LLM graders, and in pre-launch comparisons where there is no objective ground truth and expert judgment is required.

Consistency in human grading must also be managed. The Google PAIR guide explains that uncovering differences in how humans interpret labels prepares model development for reality. Two evaluators grade the same sample independently to check agreement, and review diverging cases together to refine the rubric. When comparing two versions, a blind comparison presenting them side by side without letting evaluators know which version is which reduces bias.

Capability evaluations and regression evaluations

Anthropic divides evaluations into two types according to their purpose:

Tasks that achieve sufficient proficiency in capability evaluations are "graduated" to the regression suite, where they are run continuously to monitor whether performance degrades. Anthropic refers to the state where an evaluation passes all solvable tasks and no longer shows room for improvement as evaluation saturation. Saturated evaluations lose discriminative power, so difficult tasks must be replenished with new failures discovered in production.

Offline evaluations and online metrics

Offline evaluations measure results on a prepared evaluation set without exposing them to users. They are fast and repeatable, but cannot reveal situations or user behaviors not captured in the evaluation set. Online metrics are metrics measured in actual production. These include regeneration clicks, handoffs to human agents, drop-offs, reports, and purchase conversions.

Anthropic explains that the most effective teams use a mix of methods: automated evaluations for rapid iteration, production monitoring to verify real-world outcomes, A/B experiments to validate major changes when sufficient traffic is available, user feedback and log reviews for areas automated evaluation misses, and periodic human reviews for calibration.

When offline scores are strong but online metrics do not move, it is a signal that situations not present in the evaluation set exist in reality. Read actual logs to identify the missing situations and add them to the evaluation set.

Succeeding once versus succeeding every time

The method of counting success when attempting the same task multiple times also varies depending on requirements. Anthropic describes the probability of succeeding at least once out of k attempts as pass@k, and the probability of succeeding in all k attempts as pass^k. If the probability of success for each attempt is identical at p and each attempt is mutually independent, pass^k is p to the kth power; thus, even with a 90% single-attempt success rate, three consecutive successes reach only about 73%. For a feature where a human selects one candidate among several (suggesting phrasing options), pass@k is the appropriate metric, whereas for a feature where customers must receive responses of consistent quality every time (automated customer support agent), pass^k is the appropriate metric. Even with the same score, launch decisions differ depending on what is being counted.

Deciding whether to launch based on evaluation results

Upon receiving evaluation results, check three things first before looking at the numbers:

1. Does the evaluation set represent reality? If it is not representative, high scores do not constitute evidence.
2. Is grading reliable? Has the LLM grader been cross-checked against humans, and is the score increase not caused by unrelated factors such as length?
3. Was it verified on a held-out set? A score increase observed only on the development set may be overfitted.

Once it passes these three checks, interpret the results by category. Even if the overall score exceeds the threshold, specific categories may fall significantly short. At that point, there are three options:

If the core value of a feature (e.g., reducing support resolution time) must be verified prior to launch, place that criterion under blocking criteria rather than improvement targets. The NIST AI RMF instructs organizations to determine whether systems achieve intended purposes and objectives, and whether to proceed with development or deployment (MANAGE 1.1). This decision must be made using criteria agreed upon before viewing results. Establishing criteria after seeing results leads to adjusting the criteria to fit the numbers.

Evaluation plan

An evaluation plan is written on a single page:

[Evaluation Plan]
Target feature and version:
Success criteria (from Chapter 2): Blocking criteria / Improvement targets
Evaluation set: Sources, size, composition by category (vs. actual ratios), edge case suite, high-risk suite, held-out sets separated from each suite
Grading: Method by criterion (code/LLM/human), LLM grader calibration method, and agreement target
Regression suite: Composition, execution timing (on every change), passing conditions
Online metrics: Metrics to observe post-launch and alert thresholds
Decision rules: Conditions for launch / conditional launch / hold
Schedule and owner:

Going deeper

Does a higher score mean it has improved?

When receiving a report that evaluation scores have increased, rule out four possibilities in order. First, the evaluator may have rewarded an unintended feature. It is common for an LLM evaluator's score to rise simply because an answer became longer. Verify this by having a human read a sample of logs directly. Second, adjustments may have been fitted to the same set. Check whether scores also increased on the holdout set. Third, the evaluation set may have changed. Adding or removing items alters the meaning of the score, so record versions. Fourth, it may be random variation. Run the same evaluation multiple times to understand the margin of fluctuation.

The increase that remains after ruling out these four is a genuine improvement. Teams that follow this sequence experience the situation where "scores went up, but customer complaints stayed the same" far less often.

The lifecycle of an evaluation set

An evaluation set is not an asset that is finished once created. OpenAI calls evaluation a journey rather than a destination and advises expanding evaluation sets over time, while Anthropic warns that saturated evaluations fail to show room for improvement. The lifecycle of an evaluation set is as follows:

1. Start with 20–50 actual failure cases and representative samples.
2. Add new failures found in production to the regression suite.
3. Graduate capability tasks that have been mastered to the regression suite.
4. Resample representative samples when the distribution of users and inputs changes.
5. Record versions so that results can be compared with previous ones.

The principle of "never changing once created" stems from good intentions to preserve comparability, but it causes testing to continue with an outdated test paper in a world where distributions have shifted. Comparability is preserved through version control, while the set continues to grow.

The attribution problem of online metrics

Just because metrics improve after launch does not mean it can be attributed to the feature. This is because seasonality, events, and changes to other features occur simultaneously during the same period. Google SRE points out that time is one of the biggest factors that alters observed metrics. A pre-and-post-launch comparison can cause weekdays to be conflated with weekends, and event periods with normal days.

To attribute an effect to a feature, groups that were exposed to the feature and those that were not during the same period must be randomly split and compared. Therefore, effects on business metrics are measured through A/B testing. If traffic is insufficient, the statistical power of the experiment is lacking; in that case, examine operational metrics, log reviews, and user interviews together, and report conclusions with lower certainty.

At work

Greenway Bikes' breakdown reporting bot

The bike-sharing service 'Greenway Bikes' built a conversational bot to receive breakdown reports. The development team evaluated it using 200 questions written by team members and reported an accuracy rate of 95%. Before looking at the numbers, the product manager checked three things.

First was representativeness. Looking at actual report logs, 35% consisted of a single photo and a short sentence such as "This does not work," but the evaluation set contained no such reports. The product manager randomly sampled 400 cases from reports over the previous two months to reconstruct the evaluation set. Next were high-risk cases. Brake failure reports accounted for 1% of the total, but those bicycles required immediate suspension of operation. Collecting 80 brake reports separately, they set "zero missing suspension-of-operation notices" as a separate passing criterion.

Grading was divided by criterion. Breakdown part classification was checked against ground-truth labels using code, while the courtesy of informational statements was scored using an LLM evaluator provided with a rubric, after first being calibrated against 100 cases graded by the maintenance team lead. As a result of the re-evaluation, overall accuracy was 88%, and the brake suite had 3 missed suspension-of-operation notices. The rejected proposal was "launch because overall accuracy exceeded the 85% threshold." Because the brake suite failed to pass its separate criterion, the team added a rule to route immediately to the suspension-of-operation procedure rather than the bot whenever brake-related expressions appeared, re-evaluated, passed, and then launched.

Companion Study's book descriptions

The publishing house 'Companion Study' had to choose between two versions of a feature that used AI to write introductory texts for new books. There were no ground-truth labels, and deciding whether an introduction was good required an editor's judgment. Because it was pre-launch, it could not be released to readers either.

The product manager first singled out the parts where code-based grading could be used. Whether the author's name, publication date, and page count matched the book information was verified with code. For the remaining criteria—"Does it refrain from distorting the core of the book?" and "Does it make readers want to read it?"—three editors compared the two versions for 60 new books side by side in a blinded state, without knowing which version was which. The presentation order was rotated for each book to reduce position bias.

The editors selected Version B as better for 41 out of 60 books, and instances of distortion were 7 for A and 2 for B. The team selected B, summarized the 180 evaluations from the three editors (60 books × 3 people) into a rubric, and decided to calibrate the LLM evaluator against those decisions before using it in the future. The rejected proposal was "an A/B test using reader click-through rates." This was because it was pre-launch, and click-through rates cannot indicate whether distortion occurred. They decided to track click-through rates after launch as a separate online metric.

Common mistakes

1. "Checking a few examples is sufficient." Evaluating based on intuition is a classic failure pattern. An evaluation set that represents the actual distribution is required.
2. "Excluding ambiguous cases makes grading clean." Ambiguous cases are boundary cases that fail most frequently in real-world scenarios. Tag them, include them, and examine them separately.
3. "If overall accuracy is high, rare types are also fine." A 0.4% high-risk type gets buried in the overall score. A separate suite and separate criteria are required.
4. "Human grading is expensive, so we do not use it." Human grading is simply not applied to everything; it is essential for baseline data and evaluator calibration.
5. "Scores from the LLM evaluator went up, so it has improved." An uncalibrated evaluator may reward unintended features like length. Read logs and calibrate against humans.
6. "Continually making fixes against the same evaluation set ensures certainty." It only becomes overfitted to that set. Make final determinations using a holdout set.
7. "An evaluation set is never changed once created." Preserve comparability through versioning, and grow the set with failures discovered in production.
8. "Metrics improved after launch, so it is thanks to the feature." A randomized comparison is required to isolate other factors from the same time period.

Summary at a glance

Check the official docs

• Claude Docs — Define success criteria and build evaluations: Evaluation design principles, edge cases, three grading methods, and tips for LLM grading.
• OpenAI — Evaluation best practices: Five steps of the evaluation process, evaluation-driven development, failure patterns, human calibration, position bias. This page also notes that OpenAI's Evals platform will become read-only on October 31, 2026, and shut down on November 30, but the principles are independent of the platform.
• Anthropic — Demystifying evals for AI agents: Capability evals and regression evals, pass@k and pass^k, pros and cons of grading methods, eval saturation, combining methods.
• Google PAIR Guidebook — Data Collection + Evaluation: Test sets, representativeness, differences in label interpretation.
• Google SRE Workbook — Canarying Releases: Control group comparisons and time factors.
• NIST AI RMF — Core: MANAGE 1.1 (determinations of mission achievement and whether deployment proceeds), MEASURE 2.4 (monitoring in operation).

Self-check

3-1. Write the five steps of the evaluation process described by OpenAI in order.

3-2. Write four edge cases that Claude documentation recommends including in evaluations.

3-3. Write the limitations of code-based grading, LLM grading, and human grading in one sentence each.

3-4. You want to evaluate the "return deadline date" and "informational tone" in the responses of a library loan guidance bot. Select the appropriate grading method for each of the two criteria and state the reasons.

3-5. Compare the objectives and expected pass rates of capability evaluations and regression evaluations, and explain when capability tasks are transitioned to the regression suite.

3-6. The development set score rose from 70% to 93%, but the holdout set score rose from 71% to 73%. What should you suspect?

3-7. State the appropriate success aggregation method for a feature where a human selects one among multiple candidates, and for a feature where customers must receive responses of consistent quality every time.

3-8. (Design) Write an evaluation plan for one AI feature you currently manage using the template from this chapter, attaching a one-line grading method and its rationale to each success criterion (blocking criteria and improvement targets).

Hands-on tasks