Chapter 4. Evaluation and observability

Learning objectives

• Design agent evaluations that distinguish tasks, trials, graders, transcripts and outcomes.
• Combine outcome and trajectory evaluations to suit the evaluation purpose, using code, model and human graders together.
• Interpret pass@k, pass^k, capability evaluations and regression evaluations according to operating conditions.
• Analyze agent failures stage by stage through traces and identify the first incorrect decision.
• Decide whether to deploy using quality, safety, cost and latency metrics together.

Introduction

The day before deploying a new version, the evaluation dashboard was green. Response quality had risen from 4.1 to 4.5, and the agent said “Processing is complete” more confidently. A week after deployment, customer support received repeated complaints: “It said it was done, but nothing happened.” The evaluation measured what the agent said, not what it did.

Agent evaluation is difficult for three reasons. The same input produces different results across runs; several correct paths can reach the same goal; and outcomes appear as changes to the world, rather than words. This chapter covers terminology, grading outcomes and trajectories, calibrating graders and interpreting repeated runs. It then addresses evaluation set management, tracing-based failure analysis and metric-based deployment decisions.

Key concepts

Basic evaluation terminology

Anthropic's “Demystifying evals for AI agents” uses these terms.

The most important distinction is between transcript and outcome. Even if the transcript says “Your reservation is complete,” the trial failed if no reservation row exists in the resulting system state.

Outcome-based evaluation

Outcome-based evaluation grades the final state produced by the agent. Anthropic says grading outcomes is usually better than requiring a prescribed sequence, because enforced sequences penalize correct solutions that are more efficient or creative. Its research system article likewise recommends focusing on the final state rather than analyzing every turn for agents that change state.

Defining outcomes takes more work than expected. Treat both user-facing artifacts—answers and report files—and system state—ledger rows and settings—as results remaining after the trial, and list them separately. Distinguish mandatory conditions from desirable ones, and decide how to count partial success, abstention and human handoff. The absence of prohibited actions is part of the outcome too. The article also recommends partial credit: scoring both an agent that completed three of four steps and one that failed immediately as zero hides improvement. Violations of mandatory conditions, however, count as failures separately from partial credit.

Trajectory evaluation

Trajectory evaluation examines the path an agent took: tool call order and arguments. Use it when outcomes alone cannot answer a question.

Google ADK covers both axes. tool_trajectory_avg_score checks exact tool-call trajectory matches, with a default threshold of 1.0. response_match_score measures ROUGE-1 similarity to a reference response, with a default threshold of 0.8. ROUGE-1 measures word overlap, so it may give low scores to correct answers phrased differently. This tendency can be greater in Korean, where particles and endings attach to words; verify thresholds on samples before using them. ADK also provides criteria for semantic matching through LLM judgment, response and tool-use quality through custom rubrics, unsupported content (hallucinations), and safety. Efficiency metrics such as token usage and tool call counts are reported but do not determine pass or fail.

Three types of graders and their combination

In practice, combine them. Code checks whether data changed; a model with a rubric checks whether a report is useful; human samples check whether that model grader is reliable. The emphasis also differs by agent type. Anthropic's evaluation article recommends deterministic grading such as passing tests for coding agents, expert-calibrated rubrics for groundedness, coverage and source quality for research agents, and checking intended outcomes such as files, database changes and orders in real or sandbox environments for agents operating interfaces.

Making model graders reliable

Model grader scores are measurements that also need validation. Calibrate them as follows.

1. Separate dimensions such as accuracy, groundedness and tone. Define observable criteria and penalty examples for each. Combining dimensions into one score hides the problem.
2. Allow “Unknown” when evidence is insufficient. Anthropic explains that this reduces grader hallucinations.
3. Create expert-graded samples and measure agreement with grader judgments. Read disagreements and revise the rubric.
4. Resample regularly to check that agreement holds.

Check commonly known biases too: favoring longer answers, preferring the first answer in a comparison, or favoring answers from the same model family as the grader. For pairwise comparisons, grade twice with the order reversed, and use examples to check that criteria apply independently of length. In Anthropic's research system, the grading setup that was most consistent and closest to human judgment used one prompt containing a rubric—factual accuracy, citation accuracy, completeness, source quality and tool efficiency—and one call returning both a 0.0–1.0 score and pass/fail. Notice that the rubric still separated dimensions even though grading used one call.

pass@k and pass^k

• pass@k: The probability of at least one success in k attempts. Meaningful when a person can choose among candidates or retries are possible.
• pass^k: The probability that all k trials succeed. Measures consistency when every operation must succeed on its first attempt.

With per-trial success probability p and independent trials, pass@k = 1 − (1 − p)^k and pass^k = p^k. In the article's example, 75% success per trial yields 0.75³ (about 42%) as the probability that all three succeed. As k grows, pass@k approaches 100% and pass^k falls toward 0%. A large gap means the agent can perform the task but cannot be relied on. Actual trials on the same task are often correlated, so use the formulas as interpretive guides and measure actual values through repeated runs.

Capability and regression evaluations

A capability evaluation asks, “What can this agent do well?” and should begin with low pass rates. A regression evaluation asks, “Can it still do what it used to?” and should maintain nearly 100% passing. Tasks that become stable at high pass rates in capability evaluations graduate to the regression suite and continue running.

When a capability evaluation passes every solvable task, it is saturated. A saturated suite cannot reveal improvement, creating the illusion of no progress despite real gains. Add harder tasks to refresh the capability suite, moving old tasks into regression monitoring. Regression judgments must also allow for probabilistic variation. Instead of alerting immediately on one failure, consider repeated trials and failure severity.

Building an evaluation set

You do not need hundreds of cases to start. Anthropic says 20–50 tasks drawn from real failures are sufficient initially. Its research system also began with about 20 queries representing real usage. Early changes have large effects visible even in small sets. Increase tasks and trials when deployment hinges on small differences, as discussed later. Good tasks have these properties.

1. Unambiguous: Two domain experts can agree on pass or fail.
2. Reference solutions: A solution confirmed to pass every grader demonstrates that the task and graders are valid.
3. Balanced: Include cases where an action should happen and where it must not. Otherwise, optimization becomes one-sided, such as searching for everything.
4. Drawn from real operations: Convert user reports and operational failures into tasks to keep the suite grounded in reality.

The article recommends first suspecting the task or grader rather than a lack of ability in the agent when a task has a 0% pass rate across repeated trials. Iterating prompts and tools against a suite can overfit it. Separate development and validation sets, using validation only for decisions.

Evaluation isolation and reproducibility

Start each trial in a clean environment. Scores inflate if files, data or history left by a previous trial make the next easier. Anthropic warns of an agent finding answers by reading Git history from previous trials. Conversely, shared resource shortages can make trials fail and deflate scores.

Record trial conditions for reproduction: model and version, prompt and tool definition versions, external inputs such as time and search results, and whether tools were mocked or connected to real systems. Mock-tool evaluation is fast and safe but does not capture real API latency, errors or changing data. Check again in a realistic environment before deployment.

Tracing

A trace represents one workflow from beginning to end. It consists of spans, units of work with start and end times. Parent-child links form a tree, with model generations and tool calls beneath an agent run.

OpenAI Agents SDK enables tracing by default and records agent runs, LLM generations, function tool calls, guardrails and handoffs as separate spans.

Documentation states that organizations using the OpenAI API under Zero Data Retention (ZDR) cannot use this tracing. For sensitive data, configure processors to retain structure while excluding or masking inputs and outputs before export.

MCP reserves traceparent, tracestate and baggage keys in request _meta for OpenTelemetry trace context propagation. traceparent and tracestate follow W3C Trace Context; baggage follows W3C Baggage. They let you connect agents and MCP servers in one distributed trace.

Start with the questions you want to answer when deciding what to record. Task, run, stage and tool call identifiers; parallel branch and handoff links; retries and approval waits; and model, prompt, tool and policy versions let you answer, “When, in which version and at what stage did this failure begin?” Anthropic describes using comprehensive tracing in production to observe agent decision patterns and interaction structure rather than individual conversation contents. Define collection scope and retention periods for observability data too.

Analyzing failures stage by stage

Agent failures usually become visible at the last stage but begin earlier. An initial date interpretation error may surface as a wrong reservation three stages later. Fixing only the last symptom allows the root cause to recur in another form. Analyze in this order.

1. Collect failed transcripts and classify by outcome: what went wrong?
2. Read each from the beginning and find the first incorrect decision.
3. Classify its stage: goal interpretation, planning, retrieval or tool selection, argument generation, tool result interpretation, verification or response.
4. Form a cause hypothesis involving the model, prompt, tool design, external environment or policy.
5. Change only one element at a time and rerun the same tasks to check the hypothesis.

Making deployment decisions with evaluation metrics

One average score hides important regressions. Structure a deployment decision table in layers.

Automated evaluation is only one defensive layer. Anthropic likens this to the Swiss cheese model. Automated evaluations give false confidence when misaligned with real use; monitoring reports incidents after they happen; user feedback is sparse and biased; and human transcript review takes time. No layer is perfect. Overlap them so they cover one another's gaps.

Going deeper

What to suspect first when scores move

When scores suddenly fall or rise, question measurement before the agent. If they fall, check whether rigid graders reject valid alternatives or external conditions—API responses and search results—changed. If they rise suspiciously, check whether traces leaked between trials or evaluation tasks were exposed during training or prompt revision. If scores do not move, suspect saturation. Whatever the direction, read transcripts directly. Anthropic emphasizes reading many trial transcripts and their grades to verify correct grader operation and fair failure judgments. Human testers of its research system discovered that early agents favored highly ranked content farms over authoritative sources, a problem automated grading alone can miss.

Consider sample size too. A four-percentage-point difference across 25 tasks may be one task's chance result. Run each task repeatedly, use the same task set across versions, and check that the difference exceeds repeated-run variability before drawing conclusions.

Evaluating conversational agents

Agents working across multiple turns, such as support agents, have many failures single-turn evaluations miss. Users may change requests midway or contradict earlier statements. Anthropic treats their success as multidimensional, combining state checks, turn limits and rubric-based tone and quality grading, and describes using another LLM to simulate users. Google ADK also provides user simulation, where an AI model generates utterances dynamically to test scenarios, and multi-turn evaluation criteria. Because conversations vary each time, simulation does not support exact reference trajectory matching or reference response comparison, but does support criteria such as multi-turn task success and trajectory quality. Evaluate simulated users too: an overly cooperative simulator makes the test easier than reality.

Turning operational cases into regressions

A fixed suite drifts away from real usage. Regularly select operational failures, human handoffs and rework cases, remove personal information, reconstruct them, attach reference solutions and graders, and add them to regression tests. Distinguish input distribution changes—more requests of a new type—from performance decline—more failures on the same type. The former signals expanding the capability suite; the latter signals fixing a regression.

At work

Maple Rental: an agent that says “complete”

At appliance rental company Maple Rental, a cancellation support agent records cancellations and books product return visits. A new version raised LLM-graded service quality from 4.1 to 4.5 out of five. But a predeployment sample check compared 200 conversations with the reservation ledger. Among conversations where the agent said the return booking was complete, 6% had no booking and 2% had two bookings for the same date.

Reading traces from the beginning revealed the first incorrect decisions. In missing-booking cases, the reservation tool rejected relative dates such as “next Tuesday” with a tool execution error (isError, Chapter 2), but the agent ignored it and declared completion. Duplicate cases followed retries after reservation tool timeouts.

The team first considered penalizing unsupported completion claims in the grading prompt, but rejected it. The grader would still see only the conversation, not the system state. Instead, a code-based outcome grader checked the ledger for an existing reservation, the date matching the customer's request and no duplicates. Completion claims contradicting state became critical failures regardless of other scores. The existing model grader continued measuring tone, separately from outcomes. The booking tool now accepted only ISO dates, and timeout messages instructed checking reservations first. Thirty reconstructed failure cases entered the regression suite.

The next version deployed with service quality 4.4 and outcome-based success 99%. The team's first evaluation principle became measuring completion claims separately from ledger state.

Ash Research: polished reports with weak evidence

The industry research company Ash Research's new report agent wrote longer, smoother reports and received much higher model-grader scores. Yet two internal researchers reading 20 samples found more statements whose cited sources did not support their claims than in the old version.

The team replaced the single score with separate dimensions: claim-level groundedness—whether the cited passage supports each sentence—coverage, source quality and writing quality. The groundedness grader could choose “Unknown: insufficient evidence.” They expanded the sample to 40 reports, had both researchers grade them, and measured grader agreement. Groundedness criteria with poor agreement were revised twice with additional penalty examples.

The deployment table required mandatory passes (unsupported claims at or below 2% and zero prohibited-source citations), comparative metrics (per-task success and pass^3), and an operating constraint: cost per report no more than 1.3 times the old version. The new version led in writing quality but failed groundedness and was not deployed. Analysis located the first error at the transition from source summaries to drafting without checking them against claims. A subsequent version added a prewriting stage pairing claims with supporting evidence sentences and passed every layer.

The rejected alternative was averaging the four dimensions and deploying if the average beat the old version. Large writing gains would mask a groundedness regression.

Common mistakes

1. “The agent said it was done, so it succeeded.” Looking only at conversation makes this seem certain, but transcripts record the process and outcomes record the state afterward. Check outcomes for agents that change state.
2. “Evaluation scores fell, so the new model is worse.” Clear numbers invite a conclusion, but a rigid grader may have failed a different, correct path. Do not conclude before reading the transcripts.
3. “High pass@k is enough for production.” A high number looks reassuring, but pass^k is closer to actual experience when operations offer neither retries nor candidate selection.
4. “An evaluation set needs hundreds of cases to matter.” Larger sets seem more trustworthy, which can delay starting. Initially, even 20–50 real failures reveal large changes. Prioritize representativeness, balance and unambiguous tasks.
5. “Model grader scores are objective numbers.” Automatically generated scores appear objective, but graders are measurement tools requiring calibration too. Validate agreement with human-graded samples.
6. “Average the metrics into one number.” This makes reporting convenient, but averages hide critical regressions. Keep mandatory conditions separate from averages.
7. “Sensitive data prevents tracing.” The risk may tempt you to disable tracing entirely, but you can retain structure while disabling input/output collection or masking it before export.
8. “A task with a 0% pass rate is beyond the agent.” That is the easiest interpretation, but first consider whether the task or grader is broken. Check with a reference solution.

Summary at a glance

• Transcripts record process; outcomes record final state. Grade state-changing agents by outcomes.
• Use trajectory checks for mandatory rules, prohibited actions and efficiency. Do not require exact matches for every task.
• Code checks state, models assess nuance, and humans calibrate model graders.
• Model graders need separate dimensions, an Unknown option and expert-sample agreement checks.
• pass@k means at least once; pass^k means every time. Independent-trial formulas are guides only.
• Capability suites start low; regression suites stay near 100%. Graduate stable tasks and refresh saturated suites.
• Start with 20–50 real failures, reference solutions, balance and separate development and validation sets.
• Use clean environments for every trial and record conditions and versions.
• Find the first incorrect decision in failure analysis and change one element at a time.
• Deploy through layers: mandatory passes → comparative metrics → operating constraints → deferred judgment.

Check the official docs

• Anthropic: Demystifying evals for AI agents — terminology, grader types, pass@k and pass^k, capability and regression evaluation, getting started
• Anthropic: How we built our multi-agent research system — small evaluation sets, LLM grading, human evaluation and production tracing
• Google ADK: Why evaluate agents — trajectory and response criteria and defaults, test files and suites, user simulation
• OpenAI Agents SDK: Tracing — default spans, disabling, wrapping and grouping, sensitive data and processors
• MCP 2026-07-28: Base protocol — trace context keys in _meta
• Anthropic: Writing effective tools for agents — tool evaluation metrics and held-out sets

Self-check

4-1. Using an agent that books reservations as an example, explain what the transcript and outcome each record.

4-2. Give one case where trajectory checks should apply and one where they should not, explaining why.

4-3. Assume a per-trial success rate of 0.9 and independent trials. Calculate pass@3 and pass^3 and explain what their difference tells you.

4-4. Describe three steps for calibrating a model grader against human judgment.

4-5. Explain the relationship between traces and spans, and name at least three span types in one agent run.

4-6. Why must you distinguish the first incorrect decision from the visible symptom in a failed transcript?

4-7. Design a deployment decision table for a new version of an operational agent. Include at least one mandatory condition, comparative metric, operating constraint and condition for deferring judgment.

Hands-on tasks