Chapter 5. Evaluation and operations

Learning objectives

• Define measurable success criteria and build evaluation sets covering real distributions and edge cases.
• Choose appropriate grading methods per criterion and calibrate LLM graders against human grading.
• Pin model versions and design regression evaluations and staged transitions before switching.
• Record logs and traces needed for incident investigation while minimizing personal information.
• Start from operational questions to define metrics, normal ranges, alerts and owner actions.

Introduction

The day a provider announced a faster, cheaper model, a company's executives ordered a switch for the next week. Developers tried ten representative questions and concluded that the answers appeared equally good. Two weeks after switching, complaints flooded customer service. The new model often omitted a key condition in rare, complex inquiries, while a parameter tuned for the previous model was silently ignored. Ten questions could not reveal these problems.

LLM features keep changing after entering production. Models behave probabilistically, providers release new models and retire old ones, and user inputs change over time. This chapter explains how to base decisions on quantitative metrics rather than the impression that “it seems to work”. It proceeds through success criteria and evaluation sets, grading and grader calibration, interpreting results, version pinning and switching, logs and tracing, personal information, operational metrics and cost management by feature.

Key concepts

Operations is instrument flying

A pilot flying through clouds watches instruments rather than the view outside. Senses can mislead; instruments faithfully show what they measure. LLM features work the same way. Judgments based on reading a few answers suffer selection bias and miss rare failures. Operations has two instrument panels: offline evaluations before changes and ongoing production metrics. Both must start from the same success criteria to remain comparable.

Define success criteria first

Anthropic says good success criteria are specific, measurable, achievable and relevant. Replace “performance must be good” with a narrower target such as “inquiry classification accuracy”, measured quantitatively or on a well-defined scale. Most features need several dimensions together.

Write a pass threshold for every criterion in advance. Setting thresholds after seeing results invites fitting numbers to the conclusion you want.

Build an evaluation set

An evaluation set is a regression test for a production feature: fixed inputs, expected results (or grading criteria) and grading methods.

1. Reflect the real distribution. Sample production logs to match proportions of task types, lengths and languages. First handle personal information using the methods covered later.
2. Include edge cases. Anthropic lists irrelevant or nonexistent inputs, excessively long inputs, harmful inputs and ambiguous inputs. Add business-specific high-risk areas such as large monetary amounts or support conversations with crisis signals.
3. Weight by failure cost. Oversample rare but harmful areas and report them separately.
4. Separate development and validation sets. Keep inputs repeatedly used for prompt tuning separate from those used only for final decisions, preventing overfitting to the evaluation set.
5. Version everything. Store inputs, expected results and grading code together in the repository and record every change.

Anthropic recommends many automatically gradable questions over a small, carefully human-graded set. Scale makes small regressions visible.

Choose a grading method

Start with the fastest, most reliable, most scalable method. Anthropic summarizes the choices as follows.

The same documentation gives example metrics by criterion: exact match to correct answers for classification, sentence-embedding similarity for answer consistency on similar questions, ROUGE-L for capturing key summary information when reference summaries exist, an LLM 1–5 scale for tone, and binary LLM judgment for personal-information presence. Pair each criterion with a method so the degraded dimension is immediately visible.

Calibrate an LLM grader

An LLM grader is convenient, but it is itself a model with errors and biases. Calibrate it before use.

1. Write a specific rubric. Define judgment conditions such as “Fail if the first sentence lacks an action timeline”.
2. Restrict outputs to fixed values such as pass/fail or 1–5. Free-text evaluations are difficult to compare at scale.
3. Have it reason before scoring. Anthropic explains that thinking-enabled graders perform better on complex judgments.
4. Grade with a different model from the generator. Anthropic's example code also recommends this practice.
5. Compare with human-graded samples. Have people grade 50–100 cases and measure agreement. If passes and failures are imbalanced, also inspect chance-adjusted measures such as Cohen's kappa or per-class precision and recall. Revise the rubric if below target.
6. Check known biases. Reverse ordering to test whether longer answers receive more generous scores or the first answer in a pair is preferred.
7. Pin grader-model and rubric versions. A changed grader changes what scores mean.

Interpret results: variation and segments

Three things are often missed when reading evaluations.

• Variation. The same input may differ between runs (Chapter 1). On 100 inputs, one result is one percentage point. A one- or two-point difference may be chance; rerun the same set to measure variation first.
• Paired comparisons. Rather than comparing two separate averages, align results for the same inputs and count where the new setup wins or loses. Inspecting cases where the new setup loses reveals causes.
• Segments. An improved average can hide harm to important segments. Report separately by task type, length and risk, with separate high-risk thresholds.

Pin model versions

Pin production apps to specific versions. OpenAI recommends specific model snapshots for consistent behavior. Anthropic's model list says that from the 4.6 generation onward, all Claude model IDs, including undated ones, are fixed snapshots; earlier aliases are convenience names pointing to dated IDs. Each model publishes a retirement boundary (“not before this date”), so pinning buys planning time rather than guaranteeing permanent availability.

Model ID alone cannot explain result changes. Manage reproducible configuration bundles.

Regression evaluation before switching models

Judge switches against predefined criteria rather than appearances.

1. Write criteria first. Overall and high-risk pass thresholds, latency/cost ceilings, stop conditions and responsible owners.
2. Compare current and candidate configurations on the same evaluation set. Use validation inputs and inspect losses through paired comparison.
3. Retune parameters. Do not carry settings over unchanged. Remeasure reasoning effort, remove settings the new model rejects or ignores (sampling, prefill, forced tool choice), and recount tokens with the new model.
4. Run in shadow mode. Send some production traffic to the candidate too, comparing without showing its output to users. Do not execute state-modifying tools.
5. Switch gradually. Start with part of traffic and monitor metrics. Roll back immediately if stop conditions are met.
6. Keep records. Save comparisons, decisions and configuration bundles as the next switch's baseline. Evaluation gates in the deployment pipeline can block changes below thresholds, applying the same procedure to prompt edits.

Logging and tracing

To investigate an incident, you must be able to locate a specific request again. The Claude API gives every response a unique request-id header and includes request_id in error bodies; Python and TypeScript SDKs expose _request_id on response objects. This ID is needed for provider support.

If one user request leads to several model calls and tool executions, per-call logs do not reveal the full flow. Assign a trace ID to each business task and record model calls and tool executions as spans beneath it. Distributed-tracing standards and generative-AI conventions such as OpenTelemetry help align formats across tools.

Do not log raw prompts and responses by default. If necessary, define short retention, access controls and masking together.

Handle personal information

Personal information is copied from user input to provider requests, logs, evaluation sets, batch files and caches. Masking one name is insufficient; manage the whole flow.

Legal principles. Article 3 of Korea's Personal Information Protection Act (effective September 11, 2026) requires collecting the minimum personal information necessary for the processing purpose, minimizing privacy intrusion, using anonymization where sufficient, and pseudonymization otherwise. Article 2 defines pseudonymization as deleting or replacing personal information so a specific person cannot be identified without additional information. Under Article 23, sensitive information such as health data may be processed only under exceptions such as separate consent. Article 26 requires written arrangements for outsourced personal-information processing and disclosure of the work and processor; Article 28-8 requires one of the statutory grounds for overseas provision, outsourced processing or storage. Separate consent is a typical ground; outsourced processing or storage needed to perform a contract can also rely on disclosing transfer details in the privacy policy or notifying the individual (paragraph 1, subparagraph 3). Article 29 requires safeguards such as internal management plans and access-log retention. Confirm actual application with privacy and legal staff.

Provider retention. Policies vary by provider and contract; check the documents.

User identifiers. Use opaque identifiers for abuse detection. Anthropic says not to put names, email addresses or phone numbers in metadata.user_id, using UUIDs or hashes instead. OpenAI recommends a safety_identifier made by hashing a username or email. This lets policy violations be traced to a user rather than the whole organization.

Define operational metrics

Metrics that are merely collected become a logbook opened only after incidents. Start with operational questions and connect metric, normal range, alert and owner action in one row.

Keep technical metrics (errors and latency) alongside business metrics (automation and human correction). If every response is fast and HTTP 200 but people rewrite it every time, the feature has failed.

Divide costs and limits by feature

When features share one account, one runaway feature can stop others, and a bill alone cannot explain why. Use provider isolation units. Anthropic allows workspace spend and rate limits below organization limits (unavailable for the default workspace; organization limits always apply). The anthropic-workspace-id response header identifies the charged workspace. OpenAI has organization- and project-level rate limits. Add feature/customer tags in logs to calculate costs per feature and customer organization.

Going deeper

Avoid dependence on evaluation tools

Evaluation data and grading criteria are company assets, but keeping them only within one platform ties them to its lifetime. There is a real example: OpenAI's documentation says the OpenAI Evals platform becomes read-only for existing users on October 31, 2026 and is scheduled to close on November 30. OpenAI is also deprecating reusable prompt objects in the dashboard and recommends storing production prompts in application code.

The lesson applies to every provider. Version evaluation inputs, expected results, grading code and rubrics, prompts and configuration bundles in a repository. Build runners that need only a provider-adapter swap. Use managed evaluation tools for execution and visualization while keeping originals in the repository. Then every model switch retains an unbroken record comparing before and after by the same standard.

Feed feedback back into evaluation sets

A failure found in production should become an evaluation case so the same failure does not recur.

1. Collect signals. Record explicit feedback (likes, dislikes, reports) and implicit signals (regeneration, human support agents rewriting most of a draft, immediately closing the window).
2. Classify. Group failures by cause: factual error, format, tone or tool misuse.
3. Create cases. After handling personal information, add expected results and include cases in the evaluation set, incrementing its version.
4. Run regularly. Even without configuration changes, periodic evaluations reveal input-distribution shifts or unexpected behavior changes early.

At work

Haedam Welfare Foundation: a higher average, more missed signals

Social welfare organization Haedam Welfare Foundation summarizes counselors' notes and flags crisis signals such as self-harm references, suspected abuse or urgent livelihood threats. It considered a cheaper model to cut costs; information-systems manager Yeom led regression evaluation.

The evaluation set contained 320 pseudonymized notes from the previous six months, deliberately oversampling 70 with crisis signals. An LLM grader scored summary quality from 1 to 5, while code compared crisis flags with social-worker labels. The grader differed from the candidate model and was used only after achieving 0.86 agreement against 60 cases graded by two social workers.

Results conflicted. Average summary quality rose from 4.1 to 4.3, but crisis-signal recall fell from 96% to 88%. Paired inspection showed the new model often missed indirect phrases such as “Lately I want to give everything up”. The predetermined high-risk threshold was 95%.

Yeom held the full switch and proposed a two-part approach: keep crisis detection on the current model and move only summarization to the new one. Missed expression types would enter the evaluation set for future candidates. Switching on average scores alone or relying only on grader scores was rejected. The partial switch cut costs 35% while maintaining crisis-signal recall.

Goun Laundry: entry codes in pickup notes

Laundry pickup app Goun Laundry summarizes customer pickup notes for drivers and classifies request types. A security inspection found that customers entered door codes and contact details in notes. Raw text remained in application logs for 90 days and evaluation CSVs indefinitely. The evaluation set was also shared with an external quality-review company.

Privacy manager Ha and development lead Gi first mapped the data flow. Notes were copied from app → order DB → model API → logs, evaluation sets and weekly batch files. They took these steps.

1. Mask at entry. Replace phone numbers and numeric strings resembling entry codes with placeholders before sending notes to the model, and check that summaries preserve placeholders. Only authorized drivers see original notes from the order DB in their app.
2. Keep metadata instead of raw text in logs: trace ID, provider request ID, model ID, token counts, classification and validation results.
3. Rebuild the evaluation set from pseudonymized samples and set retention periods. Retrieve and delete old CSVs and shared copies; add a CI check that fails if personal-information patterns appear in evaluation sets.
4. Review provider settings and contracts. Disable conversation-state storage, update outsourcing details in the privacy policy and review overseas-transfer implications with legal staff.

A system instruction saying “Do not output personal information” was rejected because it did not address inputs copied into logs and evaluation sets. Disabling all logs was rejected because it would prevent incident tracing. A month later, reinspection found original personal information in only one location: the production database.

Common mistakes

• An alias pointing to the latest model improves automatically. Behavior can change without validation. Pin production versions and evaluate switches.
• Pinning a version means using it forever. Models retire. Pinning buys time to plan replacement.
• A few representative questions are enough. They miss regressions in rare high-risk and edge cases. Use a fixed set reflecting the distribution.
• A higher average means a switch is safe. Important segments may degrade. Define separate thresholds per segment.
• An LLM grader can be trusted as is. Calibrate against human samples and manage biases and versions.
• Keep all raw text for a long time for debugging. This increases privacy risk. Prefer metadata and retain raw text only within necessary scope and duration.
• No training use means no privacy issue. Retention, abuse monitoring, outsourcing, overseas transfers, and your own logs and evaluation sets are separate issues.

Summary at a glance

• Success criteria: specific, measurable, achievable, relevant. Set thresholds before seeing results.
• Evaluation sets: real distribution + edge cases + oversampled high-risk segments; separate development/validation; version in the repository.
• Grading: prefer code, then LLM, then people; calibrate LLM graders against human samples.
• Interpretation: measure variation, compare paired results, aggregate by segment.
• Versions: fixed snapshots plus configuration bundles. All Anthropic IDs from generation 4.6 onward are fixed snapshots.
• Switching: criteria first → same-set comparison → retune → shadow → staged transition → records.
• Logs: trace ID, provider request ID, configuration version, usage and stop reason. No raw text by default.
• Personal information: minimum collection and pseudonymization, provider retention checks, outsourcing/overseas-transfer review, opaque identifiers.
• Metrics: question → metric (denominator) → normal range → alert → owner action. Include business metrics.

Check the official docs

• Anthropic — Define success criteria and build evaluations: read success criteria, evaluation design, grading methods and LLM-grading guidance.
• Anthropic — Models overview: check model IDs, snapshots, retirement dates and the Models API.
• Anthropic — API and data retention: check ZDR scope and feature-specific applicability.
• OpenAI — Data controls in the OpenAI platform: read training use, retention, ZDR and data-residency locations.
• National Law Information Center — Personal Information Protection Act: read Articles 2, 3, 23, 26, 28-8 and 29.

Self-check

5-1. List four qualities of good success criteria, then rewrite “Answer quality must be good” as at least two measurable criteria satisfying them.

5-2. List the calibration steps required before putting an LLM grader into production evaluations, in order.

5-3. Describe Anthropic model IDs from generation 4.6 onward and aliases of earlier models, and explain why pinned versions still require replacement planning.

5-4. State the Personal Information Protection Act's definition of pseudonymization and explain its application to evaluation sets for an LLM feature.

5-5. A candidate model improves the evaluation-set average but worsens the high-risk segment. Describe a decision procedure and possible choices.

5-6. Choose an operating LLM feature, define five operational questions and design a table giving each question's metric (including denominator), normal range, alert condition, owner and first action.

Hands-on tasks