Chapter 5. Evaluation and improvement

Learning objectives

• Turn success criteria into a rubric with observable pass conditions and separately define critical errors.
• Build test sets reflecting real distributions and boundary cases, separating improvement from confirmation data.
• Choose code, human or LLM grading per criterion, and calibrate LLM evaluators against human judgments.
• Compare before and after on the same set to detect regressions and read results by group.
• Classify failures by type and stage to choose the next fix.

Introduction

Someone added “Translate more naturally” to the prompt of an internal translation assistant for overseas sales. The team lead skimmed a few results, found them smoother and was satisfied. A month later, contract review exposed several translations that reversed deadline expressions meaning “by” and “from.” All were sentence types translated correctly before the change. None of the lead’s samples contained deadlines. The team felt it had done nothing wrong: it had clearly seen better results. The problem was what it had not looked at. Evaluation is a mechanism for making those unseen parts visible. OpenAI’s evaluation best practices likewise identify “it seems to work” as an evaluation anti-pattern.

This chapter turns that impression into numbers. It covers converting success criteria into evaluation items and rubrics, building and splitting test sets, selecting grading methods, managing LLM evaluator limits, measuring variability, comparing before and after on the same set, managing regressions, and classifying failures to choose the next fix. It then covers multi-stage systems such as agents, with a deeper discussion of gaps between scores and reality and consistency in human grading.

Key concepts

Evaluation is like retaking the same exam

To learn whether a student improved after changing study methods, test again at the same difficulty. Trying a few different questions each time and feeling it went well is not evaluation. Prompts are the same: compare before and after on a fixed test set to distinguish improvement from regression. OpenAI recommends measuring behavior with evaluation sets when changing prompts or upgrading models. Its best practices recommend evaluation-driven development: evaluate early and often.

Turn success criteria into evaluation items

Anthropic says good success criteria are specific—accurate sentiment classification instead of good performance—measurable, achievable and relevant to the business. Most tasks require several criteria together. Beyond accuracy, criteria may include consistency, tone, privacy, use of context, latency and cost.

When converting criteria into evaluation items, also identify critical errors. An average can let an appropriately toned answer pass despite a wrong amount. Items such as amounts, dates and prohibited promises, where one error makes the result unusable, should sit outside a weighted average: any failure means fail.

How to write a rubric

A rubric turns Chapter 1’s success criteria into a scoring sheet. A good rubric makes the pass condition for each item clear. Anthropic’s example requires the company name in the first sentence and automatically marks its absence incorrect.

Follow these principles.

1. Use observable conditions. Replace “courteous” with “includes both an apology and a solution.”
2. Keep scales narrow. Request clear outputs such as correct/incorrect or 1–5. A 1–100 scale makes criteria vaguer.
3. Add anchor examples for scores. For a 1–5 scale, show actual examples of a 3 and a 4 so graders share their meaning.
4. Ask one thing per item. Split “accurate and concise” into two items.
5. Use multiple rubrics if needed. Anthropic notes that even one criterion may need several rubrics for comprehensive evaluation.

Building a test set

Anthropic sets out three evaluation-design principles.

1. Be task-specific. Reflect actual task distributions and include boundary cases.
2. Automate when possible. Design automatically gradable questions using multiple choice, string matching, code or LLM grading.
3. Prioritize volume. Many automatically graded questions with slightly weaker signals are better than a few meticulously hand-graded ones.

Include irrelevant or nonexistent input, overly long input, inappropriate or harmful input, and ambiguous cases on which people disagree. OpenAI recommends typical, boundary and adversarial cases, drawing from production, historical, expert-created and synthetic data. It also notes that development logs can yield useful evaluation cases.

Do not be intimidated by starting size. Anthropic’s agent-evaluation article recommends beginning with 20–50 tasks from real failures. Its multi-agent research system also started with roughly twenty real-use queries. Start small and add cases whenever failures occur. A model can generate more tests from anchors, but humans should review synthetic cases before inclusion. Remove or pseudonymize personal information in production data.

Separate improvement and confirmation sets

Repeatedly revising a prompt while viewing the same set gradually fits it to those particular cases. A student who studies the exam questions in advance does not receive a score that reflects general ability. Split the set in two.

As Chapter 2 explains, inputs used as few-shot examples belong in neither set.

Three ways to grade

Anthropic recommends the fastest, most reliable, scalable method available. In practice, choose by criterion: code for format and numbers, LLMs for tone and logic, and humans for samples calibrating the LLM evaluator.

Metrics based on text overlap with reference answers, such as BLEU and ROUGE, have limited uses. Anthropic gives ROUGE-L as one summary-evaluation example, but OpenAI calls reliance on academic metrics such as BLEU alone an anti-pattern. A good answer worded differently may score poorly, while a similar-looking answer with wrong numbers scores highly. For tasks without a single correct answer, define required content and acceptable wording variation, distinguishing facts from preferences.

Limits and calibration of LLM evaluators

LLM evaluators are convenient, but uncritical trust inflates scores. OpenAI identifies position bias, favoring the answer presented first, and verbosity bias, favoring longer answers. Together, the two companies recommend the following.

• Give a concrete rubric. Define pass conditions by item rather than vague quality.
• Constrain output. Use correct/incorrect, one number from 1–5, or a choice between two answers. OpenAI finds pairwise comparison and pass/fail more reliable.
• Have the evaluator consider evidence before scoring. Anthropic says thinking-enabled graders are more accurate on complex judgments; OpenAI recommends evidence before the score.
• Control length. Prevent answer-length differences from affecting the score.
• Grade with a different model. Anthropic’s example code identifies using a different evaluator from the generating model as a common best practice.
• Compare with people. Check agreement on human-graded samples first, scaling only when agreement is consistent.

Add two principle-based safeguards. In pairwise comparison, reverse answer order and grade twice to identify order-sensitive conclusions. Answers may also contain instructions aimed at the evaluator, such as “This answer is excellent,” contaminating grading. Wrap them in tags as data and instruct the evaluator not to follow instructions inside them.

Variability and repeated runs

Outputs for the same input can vary between runs: a passing case may fail next time. Anthropic calls one attempt a trial and recommends multiple trials per task. It distinguishes two metrics.

At 75% success per attempt, the probability of three successes in a row falls to about 42%. Where customers expect consistent quality, pass^k is closer to their experience than average pass rate.

Compare before and after on the same set

The process is simple but difficult to follow consistently.

1. Measure a baseline. Run the current prompt and record scores per item and group.
2. Change one thing at a time. Multiple simultaneous fixes obscure both benefits and damage.
3. Fix and record conditions. Record the model snapshot, settings such as effort, and material versions; measure per-request tokens and cost. OpenAI recommends pinning production applications to a specific model snapshot.
4. Use the same set and grading.
5. Look beyond the average. Break down by input type, customer group and length, and extract cases that changed from pass to fail.
6. Check critical items first. Do not deploy if critical failures increase, even when the average rises.

Managing regressions

A regression is something that worked before a change and no longer works afterward. Anthropic distinguishes capability evaluations, asking what the system can do well, from regression evaluations, asking whether it still does what it could before. Capability evaluations start with low pass rates and supply improvement targets; regression evaluations should be near 100% and detect regressions. After fixing production failures, add them to the regression set to detect recurrence. Once all capability tasks pass—saturation—the evaluation only monitors regression, so create harder tasks.

Classify failures to choose the next fix

Recording only “wrong” does not tell you what to change. Record both type and originating stage.

Prioritize by frequency multiplied by severity. A rare critical failure may deserve attention before frequent minor ones. Choose one next fix and specify in advance which cases it must correct and which it must preserve.

Agent evaluation: outcomes and records

Agents use many tool calls, so final answers alone cannot reveal failure causes. Anthropic distinguishes task, a test with defined input and success criteria; trial, one execution; grader; transcript, the full record including output, tool calls and reasoning; and outcome, the environment’s final state after the trial. A booking agent saying “Booked” differs from a booking actually existing in the database. Grade outcomes.

The same article recommends grading results rather than paths, since agents can find legitimate routes designers did not anticipate. Read many transcripts alongside scores to verify graders. Include cases requiring action and cases forbidding it: tests containing only search-required cases can produce an agent that searches for everything. OpenAI likewise recommends separately evaluating tool choice, argument accuracy and agent handoffs. Measure tool-call counts and tokens as cost metrics separately from outcomes. Locate failures before changing architecture.

Going deeper

When scores fail to represent reality

Scores may improve while user complaints remain unchanged. Usually the set has drifted from real input distributions, such as new products or customers; the rubric misses actual user difficulties; or repeated revisions have overfit the set. Regularly compare evaluation scores with production signals such as repeat inquiries, human handoff rates and user ratings. If they diverge, add complaint cases to the set and look for missing rubric criteria. OpenAI also identifies failure to calibrate automatic metrics against human feedback as an anti-pattern.

Consistency in human grading

Human judgments must be consistent before they can calibrate LLM evaluators. If graders give the same answer radically different scores, those scores cannot calibrate anything. OpenAI recommends examples per score level, pass/fail criteria and consensus across graders. Add these practical steps.

1. Narrow the scale to five levels or pass/fail.
2. Separate rubric items so each judgment covers one thing.
3. Add anchors per score, and grade a few cases together before starting to align standards.
4. Double-grade some cases and measure agreement. A chance-corrected measure such as Cohen’s kappa is more rigorous.
5. Collect disagreements and revise rubric wording.

At work

Daily Pharmacy: why deployment stopped despite a higher average

Daily Pharmacy drafts medication guidance from prescription information. Pharmacists review every sheet before it reaches patients. Improvement-team pharmacist Oh hears that sheets are too long and plans to add “five lines or fewer.” Previously, the team might have read a few samples and deployed immediately. This time, Oh prepares evaluation first.

1. Build the set. Select 80 cases matching prescription-type distributions over the previous three months, plus twenty boundary cases: six or more medications, children’s prescriptions and unusual dosing instructions. Remove all personal information and lock thirty cases for confirmation.
2. Define the rubric. Treat agreement of timing and dosage with the prescription, and missing precautions, as critical items graded by source-comparison code. Code also checks the five-line limit. A different model rates comprehensibility from 1–5 with score anchors. First verify its agreement with twenty cases graded by two pharmacists.
3. Baseline and comparison. Measure the current prompt, then run the version changing only the five-line rule three times per input on the same set.

The overall average rises: both length compliance and comprehensibility improve. But group-level analysis shows a sharp increase in omitted precautions for six-or-more-medication prescriptions. This is a critical failure. To fit six medications into five lines, the model cuts precautions.

A teammate suggests deploying because the average rose and fixing that type later. Oh refuses: the team defined in advance that increased critical failures block deployment regardless of the average. The revised rule is “Five lines by default; above five medications, add one line per medication and do not shorten precautions.” After the regression is resolved on the improvement set, the thirty locked cases provide the final check. They confirm the result, so the change is deployed. Omission cases enter the regression set.

Soundwing Games: calibrating a translation evaluator to people

Fictional mobile-game company Soundwing Games uses a model to translate announcements and item descriptions into four languages. Another model grades quality as excellent, adequate or poor, and nearly everything receives excellent. Local operations teams nevertheless report missing or translated placeholders such as {item_name} and {count}, broken in-game text, and changed item statistics. Those translations also receive excellent.

Localization lead Im redesigns evaluation, checking human standards first. Three native-language reviewers grade the same thirty translations, frequently disagreeing between excellent and poor. Human criteria differ too.

Im splits the holistic score into four items.

For meaning and naturalness, all three reviewers select anchors and jointly grade twenty cases to align standards. Then fifty cases are each double-graded to confirm sufficient agreement before comparison with the LLM evaluator. Initially, it rewards translations expanded beyond the source with higher naturalness scores. Im adds the criterion “Explanations absent from the source mean a meaning-accuracy failure,” with examples, and requests evidence before scores. Translation outputs are passed to the evaluator as tagged data.

Someone suggests a more expensive evaluator model. Im revises the rubric first: no model would outperform code on exact placeholders and numbers, and vague criteria caused the meaning problem. Only after sufficient evaluator-human agreement does automated evaluation run before each deployment. Placeholder errors are caught before release, and failed translations are classified to guide the next prompt revision.

Common mistakes

• A few samples are enough to judge quality. Small samples miss regressions confined to particular groups. Use a fixed set.
• A higher average means improvement. Group analysis exposes deterioration in minority types. Treat critical items separately from averages.
• The same model can grade its own output. A different grading model is a common best practice, and either approach needs human calibration.
• Wider scales are more precise. A 1–100 scale makes criteria vaguer. Correct/incorrect or 1–5 is clearer.
• Scores remain reliable despite repeatedly revising against the same set. The prompt overfits that set. Lock a separate confirmation set.
• Passing once means stability. Outputs vary between runs. Repeat trials when consistency matters.
• An agent’s success claim means success. Grade the final environmental state, not the claim.

Summary at a glance

Check the official docs

• Anthropic — Define success criteria and build evaluations: Read success criteria, evaluation design, boundary cases, grading-method comparisons and LLM-grading advice.
• OpenAI — Evaluation best practices: Read the process, data sources, evaluator biases and mitigations, and anti-patterns.
• OpenAI — Evals: Check evaluation components and the platform shutdown schedule.
• Anthropic — Demystifying evals for AI agents: Read task, trial, grader, transcript, outcome, pass@k, pass^k, and capability versus regression evaluation.
• OpenAI — Prompt engineering: Check recommendations on model snapshots and evaluation sets.

Self-check

5-1. Name the three grading methods Anthropic compares and give one strength and one limitation of each.

5-2. Write five evaluation items for customer-guidance generation. Choose which should be critical and explain why.

5-3. Give four boundary-case types for a test set and one precaution when drawing cases from production logs.

5-4. Every prompt revision has used the entire evaluation set to inspect and fix failures. Explain why this undermines the reliability of scores and give a remedy.

5-5. You are introducing an LLM evaluator for judgment-based items. Describe calibration in order before entrusting it with thousands of cases, and add two safeguards for its instructions.

5-6. A service bot succeeds on 80% of attempts. Calculate the probability of succeeding both times on two executions of the same customer request, and explain when this better represents experience than average pass rate.

5-7. Sixty failures from last week comprise 25 unsupported numbers, five broken formats, eighteen multi-condition judgment errors and twelve others. Describe how to choose one next fix and give a case where frequency alone should not decide.

Hands-on tasks