Chapter 5. Operations and improvement

Learning objectives

• Build review checklists suited to work risks and choose full or sample review according to error severity.
• Define four operational metrics—quality, efficiency, adoption and limits—with denominators and collection methods.
• Classify failures by stage and improve instructions and materials through reproduction and regression testing.
• Handle changes in instructions, materials, connections, models and products through one change-management process.
• Measure benefits using baselines and total operating costs, and plan manual fallback and retirement.

Introduction

Three months after introducing a customer-reply drafting assistant, a team lead wrote “monthly usage tripled” in the quarterly report. Yet support agents were leaving work at the same time as before. Investigation showed they rewrote almost every draft; usage had grown because they regenerated repeatedly until they got something acceptable. The numbers looked good, but work had not decreased. Worse, a change to the free-shipping threshold made two months earlier had not been reflected in the assistant's reference materials. Since agents rewrote the drafts anyway, nobody noticed. What you do not measure cannot be improved, and measuring the wrong thing sends improvement in the wrong direction.

Automation begins on rollout day. Materials age, work changes and products change too. This chapter covers reviewing results, measuring actual operating conditions, turning failures into improvements, safeguards during changes, and demonstrating benefits. It also covers continuing work when automation stops or loses its purpose. The Chapter 1 success criteria and Chapter 2 test question set are used again here.

Key concepts

Review checklists: choosing items by risk

Without predetermined checks, human review becomes a quick skim. But using the same long list for every task buries what matters. Build checklists by choosing categories that fit the work's risks.

Start with completion conditions in the Chapter 1 step table. The checklist is not fixed: repeated reviewer corrections identify both the next checklist item and the next improvement target.

Error severity and full versus sample review

Errors do not all carry equal weight. Treating a misplaced comma and an email to the wrong customer as equivalent prevents metrics from reflecting real risk. Classify errors by severity.

Match review methods to severity. Maintain full review—checking every case—for steps capable of critical errors. Use sample review—randomly selecting some cases—for steps capable only of minor errors, saving human time. Return to full review if significant or worse errors in samples exceed the threshold. Define these switching conditions in advance.

Aligning reviewers' judgments

With multiple reviewers, judgments of the same output diverge. If one says “needs revision” and another “pass,” quality metrics depend heavily on the reviewer. Align judgments as follows.

1. Have reviewers independently assess the same 10–20 outputs without seeing each other's judgments.
2. Collect disagreements and discuss the reasons.
3. Record agreed criteria in the checklist with short examples of pass, revise and hold.
4. Repeat quarterly and whenever reviewers change.

Persistent disagreement signals ambiguous criteria. Break such items into more specific checks or replace subjective judgment with verifiable facts.

Four operational metrics

One metric alone misleads. More users with declining quality is a problem; good quality with no users has no impact. A limits metric suddenly approaching zero is not necessarily positive: the system may have stopped handing off cases it should. Products may provide metrics. Gemini Notebook shows daily user and question counts when a notebook is shared with at least four people and has had conversation activity within seven days. These indicate adoption, not quality.

Metric definitions and denominators

An “approval rate” calculated differently across teams cannot be compared. Write a definition sheet for each metric.

[Metric definition sheet]
Metric: rate approved without edits
Formula: (drafts approved without edits) ÷ (drafts with completed human review) × 100
Include: drafts whose review finished within the period
Exclude: drafts awaiting review, test drafts
Partial edits: changes to facts, conditions or promises count as 'revised'; typo/spacing-only changes count separately as 'notation edits'
Collection: download approval-screen records weekly
Owner: operations lead

The denominator matters. If 100 drafts are generated, 60 reviewed and 45 approved without edits in the same period, the rate is 45% using generated drafts and 75% using reviewed drafts. The quality of 40 unreviewed drafts is unknown, so use reviewed drafts as the quality denominator and report review coverage separately. Also define how to count partial success and holds in advance.

Choosing an evaluation method

There are three broad evaluation methods. Anthropic recommends choosing the fastest, most reliable and scalable method available.

For AI grading, supply a specific rubric, require a fixed answer such as correct/incorrect or 1–5, and first compare with human judgments to establish reliability. Different criteria within one task can use different grading methods. Evaluation cases should reflect the distribution of real work and include exceptions.

Classifying failures by stage

When you find a failure, editing instructions is tempting, but not every failure is an instruction problem. Identify the stage first.

Model replacement is not the first remedy in any row. A new model still answers from the same outdated material. Deleting conversation history does not make an assistant “forget” errors either. Instructions, materials and connections determine its behavior. OpenAI also explains that refining instructions and adding examples often solves problems faster than adding more tools.

Turning one failure into an improvement experiment

A temporary fix to one visible failure lets similar failures return in another form. Turn it into an improvement experiment as follows.

1. Reproduce it. Check whether the same input produces the failure and record the instructions, materials and settings in effect.
2. Form a hypothesis. Use the failure-stage table to choose one cause.
3. Change only one thing. Multiple simultaneous changes hide which one helped.
4. Run regression tests. Run the failure case together with the existing test question set to ensure the fix has not broken previously correct answers. In software testing, a regression occurs when previously working behavior becomes incorrect again.
5. Add the failure to the test set. Check it during future changes too.

Change management

Instructions are not the only thing that changes automation output. Five things change.

Use the same process for changes.

1. Record the change and reason; identify affected users and work.
2. Compare before and after using a fixed test set.
3. Apply to some users first and observe metrics (staged rollout).
4. Apply to everyone, keeping the previous state available for rollback.
5. Tell users briefly what changed and why.

OpenAI explains that the GPT editor's version history supports viewing and restoring earlier versions. If a product lacks versioning, store original instructions as separate dated documents. For changes outside your control—models, products and connection providers—monitor announcements and rerun the test set.

Measuring benefits

Benefits require comparison with a baseline: a baseline is a value measured the same way before adoption. Costs must include all human time, not just subscriptions or usage fees.

Net savings (monthly) = (time per case before adoption − total time per case after adoption) × monthly case count
                       − monthly maintenance time (material updates, instruction edits, test-set maintenance, training)
Total time per case after adoption = generation/input time + review/edit time + rework time

For example, with 15 minutes per case before adoption, two minutes generating and six reviewing/editing afterward, 300 cases monthly and eight hours of maintenance, net savings are (15−8)×300 = 2,100 minutes (35 hours), minus eight hours, or 27 hours. Report tool costs alongside this. Check that quality remains comparable. Less time with more errors moves costs elsewhere rather than saving them.

User onboarding and support

Users commonly drift back to old methods after a tool rolls out. The reasons often lie outside the tool.

1. First-use guidance: provide three concrete examples of work to try first.
2. Trust boundaries: explain what can be used directly and what must always be verified.
3. Reporting channel: make it possible to report an incorrect answer in ten seconds.
4. Find hidden manual work: interview nonusers and former users. Unmeasured work, such as reformatting results while moving them into another program, often explains abandonment.

Incident response, manual fallback and retirement

Plan in advance for automation stopping or behaving incorrectly.

Anthropic also recommends checking every scheduled-task result and pausing or deleting unused tasks. Unused automation that keeps running consumes money and retains outdated permissions.

Going deeper

Comparison traps

The easiest demonstration of benefit is “three months before versus three months after.” But if workload, difficulty, staff or season changed in between, you cannot know whether automation caused the difference.

Even concurrent comparison misleads when groups handle different work mixes. Make a habit of comparing by task type. An improving overall average can conceal deterioration in difficult types.

Questioning attractive numbers

When numbers look good, ask:

1. What is the denominator? Regeneration increases usage counts. Check whether the metric counts cases or attempts.
2. Whose voices are represented? Satisfaction surveys often capture continuing users, omitting reasons people stopped.
3. Are there enough observations? Twenty results in a week are insufficient to claim a zero error rate confidently.
4. What decreased? Generation time may fall while review and rework grow.
5. Where did difficult cases go? If handoffs plunge, check whether the system is forcing through work it should hand off.

Retirement is part of design

Automation need not run forever. Retirement is correct when the work disappears, a better method emerges, or net savings fall below maintenance costs. Retain a record of what ended and why, final instructions and test sets, confirmation of disconnected tools and removed sharing, and the human procedure that resumed. That record avoids relearning everything if the same idea is tried later.

At work

Ripple Culture Foundation: after the source structure changed

Every Thursday afternoon, Ripple Culture Foundation's communications team uses a scheduled task to draft announcements for the following week's cultural events. It reads a schedule spreadsheet maintained by the events department, writes each announcement and saves it to the team drive. Communications officer Im Seoyoon reviews the drafts and posts them to the website and newsletter.

One Thursday, Im found missing venues in nine of fourteen drafts. Review caught them before publication. Im first paused the following week's task and wrote this week's announcements manually using the old method. A documented fallback procedure made half a day sufficient.

Im did not begin by blaming instructions. The five change categories guided a comparison with last week. Instructions were unchanged, connections worked, and no provider model or product changes had been announced. In the change-management table, “materials” had changed; in the failure-stage table, “input” had failed because the incoming format changed. The events department had split “Venue” into “Venue (building)” and “Venue (room),” so the task could no longer find the old column.

A colleague suggested adding “Always include the venue” in bold to the instructions. Im rejected that: without venue data in the input, emphasis could prompt invented venues or borrowing one from a similar event. Instead, Im made three changes. Instructions reflected the new columns and required “Venue confirmation needed” at the top of announcements missing venue information. New test cases were added for events without venues and events with only a room. The materials register's change-contact field recorded an agreement that the events department would notify communications before changing spreadsheet columns.

Resumption required passing regression tests and the team lead's verification. After the 20 existing cases and four new cases all behaved as expected, the task resumed. Im's change record said, “The instructions had not changed. The material's structure had.”

Grass Leaf Co-op: why people were not using it

Grass Leaf Co-op's customer center introduced an assistant for drafting replies to member inquiries four months earlier. Operations specialist Namgung Hyeon gathered four metrics for the quarterly review. Adoption was the first obstacle: only six of fifteen agents used it consistently each week. Among those six, 70% approval without edits was reasonably good.

Namgung met each of the nine nonusers. Quality was not the problem. Drafts included headings and bullets in every paragraph, but the symbols broke when pasted into the support system's reply field. Agents spent two or three minutes per case deleting symbols and reformatting lines, and felt writing from scratch was easier. No metric captured this time, because efficiency measured only draft generation.

Namgung changed two things. Format instructions became “Write plain sentences that can be pasted directly into the support system's reply field. Do not use headings or bullets,” and the same test set checked that only format changed. Efficiency was redefined as time from opening an inquiry to sending its reply.

Quality metrics also needed work. Two reviewers had judged independently; when both separately assessed the same fifteen drafts, they disagreed on five. They discussed these, aligned criteria, and attached pass/revise/hold examples to the checklist.

Two months later, thirteen agents used the assistant weekly, and total inquiry time fell from eleven minutes before adoption to seven minutes. Namgung's report placed all four metrics alongside one another instead of highlighting usage counts, and ended by stating which metric definitions had changed and why.

Common mistakes

1. “Adding emphasis will fix it.” “Never be wrong” and bold type do not fix causes. Identify and address the failure stage.
2. “Changing models solves everything.” Outdated material, changed inputs and ambiguous instructions cause errors with any model.
3. “Deleting failed conversations prevents recurrence.” Deleting history does not retrain the assistant. Failures are valuable test material.
4. “Usage grew, so it succeeded.” Regeneration also increases usage. Examine quality, efficiency, adoption and limits together.
5. “We changed nothing, so results stay the same.” Materials, connections, models and products may change without your intervention. Monitor notices and rerun tests regularly.
6. “Roll the fix out to everyone immediately.” Skipping test-set comparison and limited rollout exposes everyone to new errors.
7. “Reduced generation time equals savings.” Subtract review, editing, rework and maintenance to find net savings.

Summary at a glance

• Choose checklist categories by work risk, starting from completion conditions and extending them with failure cases.
• Classify errors as minor, significant or critical. Fully review steps capable of critical errors; sample the rest. Define switching conditions in advance.
• With multiple reviewers, have them judge the same cases independently to align criteria.
• View quality, efficiency, adoption and limits together. Define formula, denominator, inclusions, exclusions and collection method for each.
• Use rules for exact values, people for ambiguous judgments, and reliability-tested AI for tone and format.
• First locate failures in input, materials, retrieval, generation, permissions, connections, review or delivery. Model replacement is not the first remedy.
• Improve through reproduction → hypothesis → one change → regression testing → addition to the test set.
• Manage instructions, materials, connections, models and products together.
• Net savings = reduced total time per case × volume − maintenance time. Check quality is maintained too.
• Define stop signals, manual fallback, resumption conditions and retirement criteria during normal operations.

Check the official docs

• Anthropic — Define success criteria and build evaluations: Read success-criteria requirements, evaluation principles, and the trade-offs of code, human and AI grading.
• Anthropic — Building effective agents: Check the principle of measuring and iterating, adding complexity only when it clearly improves outcomes.
• OpenAI — Creating and editing GPTs: Read about preview tests, version history and restoration, and the custom GPT retirement schedule.
• Google — Create a notebook in Gemini Notebook: Check notebook analytics and their conditions.
• Google — About the transition from Gems to skills: Check the product transition schedule and migration method.
• Anthropic — Use Claude Cowork safely: Read recommendations to check scheduled-task results and clean up unused tasks.

Self-check

5-1. Explain the problem with one long review checklist for every task. From this chapter's table, choose one essential category and one that can be omitted for internal weekly meeting summaries.

5-2. Give one example each of minor, significant and critical errors, and propose a condition for returning from sample to full review.

5-3. Two reviewers frequently disagree. Write three steps to align their criteria and explain how to revise items that still produce disagreements.

5-4. In one week, 200 drafts were generated, 120 reviewed and 90 approved without edits. Calculate the rate using generated and reviewed drafts as denominators. Which should be the quality metric, and what should be reported alongside it?

5-5. Classify these failures by stage: (A) expired connector authentication prevents retrieving yesterday's email; (B) the correct evidence document is found but an exception clause is omitted; (C) a changed application form prevents reading the applicant contact field.

5-6. Calculate monthly net savings from these invented figures: 500 cases, ten minutes per case before adoption, one minute generating plus four reviewing/editing afterward, and ten hours of monthly maintenance. Show your calculation and name one thing to check before calling this a success.

5-7. Name three things to decide in advance for when automation must stop and people take over. Explain the confusion caused if one is missing.

Hands-on tasks