Chapter 5. Launch and Operations

Learning objectives

• Define the launch scope (user group, feature scope, excluded types) and choose an appropriate launch method among shadow testing, canary, beta, and A/B testing.
• Set abort criteria and rollback methods before launch, and evaluate canary results through sample size and comparison with a control group.
• Design operational metrics and alerts by tiering them into system, quality, user behavior, and business metrics.
• Manage changes to models and prompts through versioning, regression evaluations, and phased rollouts, and prepare for model lifecycle notices.
• When an incident occurs, respond in the order of containment, notification and correction, root-cause fix and recovery, documentation and blameless postmortem, and measures to prevent recurrence, and prioritize improvements using operational data.

Introduction

On the day an AI feature launches, the team is usually relieved. It passed evaluations and met the launch criteria. However, the evaluation set reflects the world up until yesterday; post-launch users ask questions not in the evaluation set, model providers change models, and the team modifies prompts. The quality of an AI feature is most clearly understood on launch day, and thereafter quietly changes if not monitored.

Anthropic explains that the most effective teams use automated evaluations, production monitoring, A/B testing, user feedback, and human review together. The NIST AI RMF, a voluntary risk management framework, recommends executing post-deployment monitoring plans and communicating incidents and errors to relevant parties, including affected individuals. Launch and operations is the work of turning these principles into schedules and numbers.

This chapter first selects launch scope and launch methods, and designs canaries and abort criteria. Next, it covers operational metrics, alerts, and model and prompt change management, and finally teaches how to respond to incidents, conduct postmortems, and prioritize improvements using operational data.

Key concepts

Launch is the beginning of measurement

Launch is not the period at the end of development, but the starting point for measuring performance in the real world. Questions that offline evaluation could not answer—namely, how real users use it, what inputs arrive, and whether business metrics move—can only be answered after launch. Therefore, a launch plan must include what to measure and until when, what numbers justify expansion, and what numbers trigger a stop.

By analogy, a launch is like opening a new bridge. Even if it passes design calculations (offline evaluation), rules are set in advance to initially restrict vehicular traffic, measure vibrations with sensors, and control access if thresholds are exceeded.

Defining launch scope

Launch scope is defined along three axes.

If only a specific type falls short of criteria in evaluation and a path exists to route that type to a human, excluding that type from the launch is a scoping decision. Because exclusion must be enforced at runtime, plan mechanisms alongside it—such as intent classifiers or rule filters—that recognize inputs of that type and route them to human agents or existing guidance instead of the model. Make the scope adjustable via mechanisms like feature flags, allowing it to be expanded or narrowed later without code deployments.

Comparing launch methods

According to AWS's description of shadow testing, the new variant receives a copy of real requests, but only responses from the existing (production) variant are returned to the calling application, while responses from the shadow variant are discarded or logged for offline comparison. Google SRE defines a canary as a partial and time-limited deployment of a service change and its evaluation, explaining that it is effectively an A/B process.

Canary design

There are five principles that product managers can take away from the canary chapter of Google SRE:

1. Start small. The impact of a defect is proportional to the traffic exposed to it.
2. Compare against a control group. A canary is a process of comparing the changed portion with the rest of the service (the control group). Because time is one of the largest factors altering observed metrics, compare against a control group over the same time period rather than comparing before and after launch.
3. Choose metrics. Metrics must be capable of revealing problems, representative, and attributable to the change.
4. Long enough, large enough. A canary must be large enough and last long enough to represent the full deployment, and receive enough traffic to process a representative sample.
5. One at a time. Do not run multiple canaries concurrently. If two changes are released together, one cannot know which side to attribute metric shifts to.

In canaries for AI features, look at quality and user behavior metrics (report rate, agent transfer rate, regeneration rate) alongside system metrics (errors, latency). This is because AI failures occur without error responses.

Abort criteria and rollback

Abort criteria are set in numbers before launch. Write them as "abort if the report rate is 0.5%p or more higher than the control group" or "abort if answers without grounded citations exceed 2%." If criteria are set after seeing results, the criteria will be picked to fit the results.

Also specify in the abort criteria the minimum sample required for evaluation. A 2%p difference from 60 conversations may be a coincidence, whereas a difference from 20,000 is not. For metrics judged by proportions, if the sample is insufficient, neither expand nor roll back, but maintain the exposure ratio and extend observation. When a threshold is exceeded with a sufficient sample, stop without debate, roll back, and investigate the cause. However, immediate abort conditions that do not tolerate even a single case—such as personal information exposure, harmful generations, or legal violations—are stopped and rolled back immediately upon discovery regardless of sample size. State which conditions are immediate aborts before launch as well. Google SRE advises that if a problem is seen, stop deployment and roll back, or page someone to resolve the issue. Pushing out an immediate fix while the threshold is breached means exposing yet another unverified change.

Rollback methods are also tested before launch. Verify whether turning off the feature flag returns the system to the previous state, and determine in advance how to rectify results that have already been processed (sent messages, executed tool calls, posted settlements). Write down in the launch plan who will do what among sending correction notices, reversing transactions, or manual reconciliation by personnel.

Conditions for A/B testing

A/B testing is the most definitive way to attribute an effect to a feature, but the conditions must be met:

1. Low risk. It must be acceptable to show the new variant to half of the users.
2. Sufficient traffic. Samples must accumulate enough to distinguish differences in target metrics from chance.
3. Target metric is fixed to one. Picking metrics after seeing results confuses random variation with an effect.
4. Guardrail metrics exist. Stop if report rates or refund rates worsen, even if target metrics improve.
5. Random assignment. Allowing users to choose for themselves compares groups with different dispositions.

Anthropic explains that A/B testing is a way to validate major changes when there is sufficient traffic. If traffic is insufficient, look at operational metrics, log reviews, and user interviews together instead of an experiment, and report conclusions with lower certainty.

Tiers of operational metrics

Operational metrics must be divided into four tiers to leave no blind spots:

Looking only at system metrics misses fabricated answers. This is because plausible incorrect answers go out without error responses. Background errors described in the Google PAIR Guide—failures where neither the user nor the system perceives an error—may not be caught by user behavior metrics either. Therefore, metrics in the quality tier are needed, such as citation rates or daily sampled automated and human grading. Conversely, failures where an agent repeatedly calls the same tool before finally producing a correct answer are not caught by quality metrics, but are caught by cost metrics such as tokens per request.

Alert design

Metrics are meaningful only when someone looks at them. Alerts define three things:

1. Threshold. What value triggers an alert when exceeded? Understand the normal fluctuation range and set it outside of that.
2. Recipient and action. Who receives it, and what do they do upon receiving it (investigate, throttle features, shut down)?
3. Urgency. Separate items needing immediate response (service outages, personal information exposure, cost explosions) from items to review on the next business day (gradual quality degradation).

Alerts alone are sometimes insufficient for costs. To guard against incidents where costs balloon in a few hours, such as agent call loops, put enforced ceilings in place, such as spend limits in the provider console. The Claude documentation explains that when a spend limit set on an organization or workspace is reached, the API returns an error.

The NIST AI RMF calls for monitoring the functionality and behavior of deployed systems (MEASURE 2.4), and recommends that post-deployment monitoring plans include mechanisms to capture and evaluate feedback from users and stakeholders, appeal and override, system decommissioning, incident response, recovery, and change management (MANAGE 4.1). Alerts are the nervous system of this plan. If there are too many alerts, no one looks at them; therefore, if one cannot answer "what do we do when we receive this alert?" for a metric, place it on a dashboard rather than as an alert.

Model and prompt change management

AI features change even when the code remains identical. Results change if a prompt is edited, if a model is swapped, or if a provider retires a model or changes an API. Therefore, manage prompts and model configurations like code:

1. Keep versions. Record the prompt, model name, and configuration as versions, and log which version answered in operational logs.
2. Run regression evaluations before rollout. Even a one-line modification can alter answers for other types. Perform case-by-case comparisons with regression suites.
3. Measure quality, latency, and cost together when replacing models. Even with the same prompt, results and costs differ when the model changes.
4. Roll out incrementally. Release major changes via canary.

Also prepare for model lifecycles. Claude documentation categorizes model statuses into Active (fully supported, recommended), Legacy (no longer updated and may be deprecated in the future), Deprecated (functional but not recommended, with a replacement model and retirement date set), and Retired (unavailable, requests fail). Public models are announced at least 60 days before retirement, and testing with a new model well ahead of the retirement date is recommended. It is not just models. According to the parameter deprecation table in the same documentation, sampling parameters such as temperature, top_p, and top_k are removed in the 2026-10-21 API version and return a 400 error if specified, and even in prior API versions, providing non-default values to models after Claude 4.7 results in a 400 error. Regularly checking provider announcements is also operational work.

Incident response

Incidents with AI features vary, such as incorrect guidance, personal information exposure, inappropriate generations, and cost explosions. The response sequence is as follows:

1. Contain the spread. Turn off the feature, divert that type to a human, or roll back to the previous version.
2. Identify, notify, and rectify affected individuals. If there remain people who might believe and act upon incorrect information, notifying them comes first.
3. Fix the root cause and recover. Verify the fixed version with regression evaluation after identifying the root cause, then reopen incrementally.
4. Compile records into an incident report. Continuously document status and actions throughout the response, and once finished, compile the timeline, impact, causes, and actions into an incident report.
5. Hold a postmortem. Analyze centering on contributing causes without blaming individuals.
6. Implement and verify measures to prevent recurrence. Assign owners and deadlines.

The NIST AI RMF calls for communicating incidents and errors to relevant parties, including impacted communities (MANAGE 4.3). Incident response does not end just because a root-cause fix has been deployed. This is because people who already received incorrect guidance still remain.

Blameless postmortem

Google SRE defines a postmortem as a record of an incident, its impact, the actions taken to mitigate or resolve it, the root causes, and the follow-up actions to prevent recurrence. Examples of conditions for triggering a postmortem include user-visible downtime or quality degradation exceeding a certain threshold, data loss of any kind, and on-call intervention (such as rollback of a release or traffic rerouting). Furthermore, it emphasizes that a blameless postmortem must focus on identifying the contributing causes of an incident without blaming individuals or teams.

Common contributing causes emerging in postmortems for AI features are as follows: a prompt change was shipped without regression evaluation; the evaluation set lacked that type; there were no alerts on quality metrics; or citations were not visible to reviewers. Measures to prevent recurrence must target these causes. Add the incident cases to the regression suite, create missing alerts, and update change procedures. When adding operational conversations to evaluation sets, mask or de-identify personal information first. Google SRE also advises that the incident commander maintain a live incident document during response and preserve those records for use in postmortem analysis.

Prioritizing improvements with operational data

As operational data accumulates, there will be many things to fix. Since improvement personnel are limited, priorities must be set. The NIST AI RMF advises prioritizing the treatment of documented risks according to impact, likelihood, and available resources or methods (MANAGE 1.2).

In practice, calculate "monthly occurrences × damage per case × proportion actually delivered to customers" for each failure type. The share filtered out by existing controls (validation steps, human reviews) is excluded in the final term.

In this table, over-refusal takes priority over "groundless guidance," which appears to be the most dangerous. However, there are exceptions not determined solely by numbers. Types involving legal or safety issues or capable of severely damaging trust are separately verified to ensure controls are adequate, even if occurrences are few. Distinguishing the impression given by the name of a type from its actual damage is the core of this judgment.

Phased rollout plan

[Phased Rollout Plan]
Feature and version:
Launch scope: User group / Included types / Excluded types and their routing paths
Launch method: Shadow / Canary / Beta / A/B, reason for selection
Phases: e.g. 1% → 10% → 50% → 100% (adjusted for traffic and risk), minimum duration and minimum sample per phase
Comparison baseline: Control group over the same time period
Abort criteria: Metric, threshold, minimum sample for evaluation
Rollback method: Flags, previous version, handling of already processed results
Operational metrics and alerts: System / Quality / User behavior / Business, recipient and action
Change management: Prompt and model version, regression evaluation conditions, frequency of checking provider announcements
Incident response: Contact tree, postmortem criteria

Going deeper

The temptation to change metrics

When target metrics do not move after launch, the temptation arises to look for other metrics. "Conversion rate remained flat, but recommendation clicks increased by 30%." This report is not false, but neither is it proof that the feature achieved its objective. If clicks increased while purchases remained flat, it may mean that the recommendations only aroused curiosity and lacked usefulness.

The Google PAIR Guidebook warns that optimizing for metrics that look good in the short term (for example, the number of shares) can be harmful in the long term, and recommends imagining what would happen to users and society if the metric were perfectly optimized. Target metrics and guardrail metrics must be decided before launch; if results are poor, investigate the cause rather than changing the metrics. Changing metrics is justified only when it is revealed through evidence unrelated to the results that the initially chosen metric does not measure the objective, and even then, document the reason for the change.

Silent quality degradation

The quality of an AI feature can degrade even when nothing has changed. As the user base expands, the distribution of inputs shifts (new regions, new product categories, new expressions), reference documents are updated leaving outdated citations behind, and the provider's model changes. The evaluation set from launch time is unaware of these changes.

There are three ways to catch silent degradation. Sample from production logs daily or weekly, score them using the same grading method, and observe trends. Compare the distribution of input types against launch time to see if new types have increased (shifts in input distribution are commonly called data drift). Observe trends in user behavior metrics (regenerations, transfers to human agents). If any of the three moves, read the logs and add newly discovered failures to the evaluation set.

The decision to turn off a feature

Not all AI features need to run indefinitely. If operational costs exceed benefits, if users do not use the feature, or if risks cannot be controlled, scaling back or turning off the feature is also an operational decision. The NIST AI RMF calls for mechanisms and responsibilities to replace, decommission, or deactivate systems that deviate from their intended use.

Turning off a feature also requires a plan. Notify users in advance, guide them to alternative paths, process data left behind by the feature (conversation logs, generated outputs) according to policy, and record why it was turned off. If the rationale behind the decision to turn it off is documented in the same format as the AI adoption assessment memo in Chapter 1 (task, throughput, solution, data and cost requirements, legal review, conclusion), it becomes possible to know what must change if the same idea is brought up again later.

At work

Sprout Bank's transfer inquiry consultation

The internet bank "Sprout Bank" prepared to launch an AI consultation service to answer inquiries regarding transfer limits and fees. Daily inquiries stood at 30,000 cases, and it had cleared all blocking criteria in offline evaluations. However, because the overseas wire transfer fee category fell short of the threshold, it was excluded from scope and routed to existing human agent paths.

The product planner chose a canary release. This was because customers saw the results and human agents could correct any errors, making it easy to roll back, and because it was necessary to observe actual customer report rates and consultation transfer rates. The rollout phases were set to 1% → 10% → 50% → 100%, with judgment conditions requiring a minimum of 3 days and 900 conversations for the 1% phase, and a minimum of 3 days and 5,000 conversations or more starting from the 10% phase. The halt criteria were "report rate of incorrect fee guidance is 0.3%p or more higher than the control group" and "answers without source citations exceed 1%." A redesign of the app's home screen scheduled for the same period was postponed until after the canary completed. This was because if the two changes overlapped, it would be impossible to attribute metric changes to either one.

On the third day of the 10% phase, answers without source citations reached 1.6% across 9,000 cumulative conversations in that phase, exceeding the threshold. The team halted expansion, turned off the feature flag to revert all inquiries to existing consultation channels, and investigated the cause. The fee regulation document had been revised, and both old and new documents were being retrieved together. Checking all logs across the 1% and 10% phases revealed 152 conversations where outdated fees had been quoted; the team sent correction notices to those customers, fixed the document index, ran regression evaluations, and reopened from 1%. They also held a postmortem. The preventive measures that emerged from the postmortem were "procedures to remove indices of previous versions upon source document revisions" and "adding document revision dates as triggers for running regression evaluations." The rejected proposal was "immediately resuming at the same percentage by inserting a sentence in the prompt to prioritize newer documents." This was before the root cause had been confirmed, and doing so would have re-exposed unverified changes. Furthermore, older documents remaining in the search index could not be reliably prevented by prompt instructions alone.

Operating Sunset TV's search assistant

The video streaming service "Sunset TV" had been operating an AI search assistant for four months. Engineering resources for improvements could take on only two tasks in the following quarter. Five candidates emerged from production data: errors improperly associating cast names, errors mixing adult content into children's profile results, response latency, refusal cases caused by failing to understand search queries, and recommendations appearing similar every time.

The product planner created a table calculating monthly harm using monthly case count, harm per case, and the proportion actually delivered to customers. Refusals were the most frequent at 20,000 cases per month, and drop-offs following refusals reduced viewing starts. Children's profile errors were 6 cases per month, but there were 2 actual exposures that breached the blocking filter, which could lead to legal and trust issues. In terms of sheer numbers, the planner selected refusals as the first task, and despite the low case count, selected the children's profile error—where controls had failed—as the second task. This was a decision distinguishing between risks that were already controlled and risks that had actually breached containment.

Change management was also overhauled. Previously, prompts had been edited directly in the production environment by personnel whenever needed. They decided to version prompts going forward, and only during regular Tuesday deployments, run a 400-case regression suite to release versions that passed to a 10% canary. They also instituted a rule to check model providers' deprecation notices during the first week of every month, and if a retirement notice appeared, run full evaluations with alternative models within that week.

Common mistakes

1. "Since it passed offline evaluation, it can be launched to 100% immediately." Real users and inputs differ from the evaluation set. Limit exposure, measure, and then expand.
2. "For a canary, you only need to compare before and after launch." Time is one of the largest factors that shift metrics. Compare against a control group over the same time period.
3. "Halt criteria can be decided after looking at the results." Criteria set after seeing the results will be tailored to the results. Establish the numbers and minimum sample sizes before launch.
4. "If a threshold is exceeded, simply fix it and redeploy right away." Halt first, roll back, and then investigate the cause. Do not re-expose unverified changes.
5. "Phasing in multiple changes together saves time." You will be unable to attribute effects. Run only one canary at a time.
6. "You can understand operational health just by looking at error responses and latency." AI failures happen without errors. Quality and user behavior metrics are required.
7. "Deprecated models continue to work after the retirement date, just more slowly." Requests sent to retired models fail. Evaluate replacements well ahead of the retirement date.
8. "Incident response is complete once the cause is fixed and deployed." It is complete only after notifying those who received incorrect guidance, documenting the incident, and executing preventive measures.

Summary at a glance

Check the official docs

• Google SRE Workbook — Canarying Releases: Definition of canaries, control group comparison, metric selection, duration and scale, one at a time.
• Google SRE Book — Postmortem Culture: Learning from Failure: Definition and conditions of postmortems, blameless postmortems.
• Google SRE Book — Managing Incidents: Live incident documentation during response.
• AWS — Amazon SageMaker AI shadow tests: How shadow testing works.
• Claude Docs — Model deprecations: Model lifecycle, retirement notice periods, API parameter deprecation.
• Anthropic — Demystifying evals for AI agents: Combination of automated evaluations, production monitoring, A/B experiments, and human review.
• NIST AI RMF — Core: MANAGE 1.2 (priorities), 2.4 (replace, decommission, or deactivate), 4.1 (post-deployment monitoring), 4.3 (incident notification).
• Google PAIR Guidebook — User Needs + Defining Success: Reward functions and the risks of optimizing short-term metrics.

Self-check

5-1. State the three properties that Google SRE requires of canary metrics.

5-2. What cannot be measured with shadow testing, and what rollout method should be used if that is needed?

5-3. Explain why a minimum sample size must be specified alongside abort criteria.

5-4. Write the four layers of operational metrics, and explain why fabricated answers are not caught by the error response rate.

5-5. Write the four stages of the model lifecycle in the Claude documentation, and describe what happens to requests in the final stage.

5-6. Write the six stages of incident response in order.

5-7. There is a type with 50 incidents per month, damage of KRW 40,000 per incident, where 90% are blocked at the verification stage, and another type with 1,200 incidents per month, damage of KRW 2,000 per incident, with no blocking mechanism. When compared by monthly damage, which comes first?

5-8. (Design) Write a staged rollout plan for the AI feature you are currently responsible for using the template in this chapter, attaching a minimum sample size to two abort criteria.

Hands-on tasks