Chapter 4. Failure Handling and Human Oversight

Learning objectives

• Distinguish between fabricated answers, refusals, formatting errors, and transient disruptions, and design preventive measures and failure responses for each type.
• Select a response among retries, queues, fallback paths, graceful degradation, and feature suspension that matches the duration of the disruption and the nature of the feature.
• Determine human intervention points (prior approval, conditional escalation, post-hoc spot checks, appeal processing) based on error reversibility, throughput, and high-risk subsets.
• Design decision rationales, modification and rejection authority, and records so that reviewers can make substantive judgments.
• Design user notices, trust calibration, feedback, and control mechanisms from the perspectives of legal obligations and user experience.

Introduction

One day, an AI customer service bot at an e-commerce mall stated, "This product can be returned for free within 60 days of purchase." The actual policy was 14 days. The bot threw no errors, responded quickly, and maintained a polite tone. Every system metric was normal, but hundreds of customers acted believing the incorrect deadline. Failures in AI features are usually quiet like this.

Traditional software failures reveal themselves through error screens. AI failures hide behind plausible answers. Therefore, planning an AI feature must include two things. First is categorizing failures by type to mitigate each one and defining the system behavior when a failure occurs. Second is determining where and how humans should intervene. OpenAI recommends having humans review outputs before they are actually used whenever possible, and emphasizes that this becomes increasingly critical in high-stakes domains. However, because humans cannot be placed everywhere, choosing where to place them is what planning is about.

This chapter first examines the four types of failure and their respective responses, and covers fallback paths and graceful degradation when disruptions become prolonged. Next, it outlines the criteria for selecting human intervention points and the authority to grant reviewers, and finally designs user notices and feedback mechanisms.

Key concepts

Failure is part of planning

Failures in AI features can be broadly divided into four types.

The Google PAIR Guide also divides errors into system limitations, context errors, and background errors. In particular, background errors refer to cases where the system is not functioning properly but neither the user nor the system recognizes it as an error, with fabricated answers being a prime example. Failures unknown to users cannot be caught through user feedback, so they must be caught through design and operational metrics. Transient disruptions are further divided by duration when determining responses. Because the response differs completely between a disruption resolved in seconds and a prolonged disruption with an unknown recovery time, the response table below expands the four types into five rows. There are also security failures such as prompt injection, where malicious input bypasses instructions. Although this course does not cover them in depth, if a feature receives external input, security review findings should be included in this response table as well.

Fabricated answers

Fabricated answers (hallucinations) refer to a phenomenon where a model plausibly generates content that is contrary to fact or does not match the provided context. Claude documentation cites three basic strategies to mitigate this:

1. Allow it to say "I don't know." Explicitly permitting the model to acknowledge uncertainty can significantly reduce false information.
2. Ground answers in direct quotes. When dealing with long documents (cited as 20,000 tokens or more in Claude documentation), have the model extract the relevant sentences verbatim first, and base its answer on those quotes.
3. Verify with citations. Require citations for each claim, and have the model retract any claim for which supporting evidence cannot be found.

Advanced techniques include inspecting reasoning processes, comparing results across multiple runs of the same query, iterating to have the model re-verify results, and restricting the model to use only information from the provided documents. The same documentation notes that while these techniques substantially reduce hallucinations, they cannot eliminate them entirely, and advises that critical information must be verified, especially in high-stakes decisions.

How fabricated answers are handled depends on the nature of the feature. For factual guidance backed by source documents, bind answers to citations and instruct the model to state that it cannot verify if supporting evidence is absent. For brainstorming tasks without factual claims, generate broadly and let a human choose. For tasks that lack evidence and where errors lead to irreversible harm—such as physical assessments that cannot be verified even with source documents—avoid building the feature altogether or redirect users to human experts.

Refusal and over-refusal

A model not answering is sometimes correct behavior. Stating "Cannot verify" in response to unfounded or risky requests is normal operation that is preferable to fabricated answers. There are two kinds of failure: over-refusal, which refuses even answerable questions, and unguided refusal, which terminates the conversation without a next action even when the refusal is warranted. Over-refusal is not captured in the proportion of incorrect answers (error rate), nor does its cause surface in accuracy metrics unless refusals are counted separately. Therefore, the refusal rate for answerable questions is measured independently. Over-refusal interrupts the user flow and offloads processing costs to support agents or telephone desks.

Over-refusal is addressed in two ways. First, reduce it. Collect cases where the model refused despite being able to answer, add them to the evaluation set, and re-evaluate after modifying the prompt or the scope of source context. Second, enable users to take a next action even when a refusal occurs. The PAIR Guide recommends focusing on what users can do after a failure and letting users take over the task when a failure occurs. Instead of "Cannot answer," write something like: "This question could not be found in the policy documents. Related menu: ○○, Connect to representative: ○○." Guidance should differ between cases where the model could not answer due to lack of evidence and cases where it did not answer in accordance with safety policies. The former opens alternative paths, whereas the latter briefly states the reason without prompting the user to repeat the same request.

Formatting errors

Formatting errors are failures to adhere to the agreed output format. If it is a human-facing screen, the screen goes blank or breaks; if the output goes to another system, the integration fails. Integration failures are more dangerous because they tend to accumulate without anyone noticing.

Claude documentation explains that using structured outputs guarantees responses adhering to a JSON schema, but notes that responses refused for safety reasons (where stop_reason is refusal) or responses truncated by reaching max_tokens may not match the schema. Therefore, the response consists of three layers:

1. Lock down the format using structured outputs or strict tool use (a setting ensuring tool names and input values follow the schema). Because casing in string enum values is not guaranteed, compare them case-insensitively.
2. Validate the received output against the schema, and do not pass it to downstream systems if validation fails.
3. Route validation failure cases to a reprocessing queue or a default fallback screen, and monitor that rate as an operational metric.

Transient disruptions and retries

Model APIs also experience disruptions. Claude documentation describes error codes such as 429 (rate_limit_error, rate limit), 500 (api_error, internal error), and 529 (overloaded_error, temporary overload). It advises retrying internal errors with exponential backoff, and notes that the official Anthropic SDK automatically retries transient failures such as connection errors, rate limits, and 5xx errors up to two times by default using exponential backoff, respecting the retry-after header if present. Exponential backoff is a method that increases the waiting interval with each retry, such as 1 second, 2 seconds, and 4 seconds.

Retrying is a response suited to short disruptions. If it is an overload resolved in seconds, retrying a few times with increasing intervals solves most issues; if it exceeds the set number of attempts, notify the user and provide an alternative path. If retries are not capped, users wait indefinitely, and requests flood the impaired server, exacerbating the situation. The same documentation also recommends ramping up traffic gradually, as sudden spikes in usage can trigger 429 errors due to acceleration limits (limits placed on surging usage).

Fallback paths and graceful degradation

When a disruption becomes prolonged or the recovery time is unknown, retries alone are insufficient. The response differs depending on the nature of the feature.

Graceful degradation is a design that allows core tasks to continue through simpler paths even when AI features stop working. If AI-recommended search goes down, it falls back to keyword search; if AI consultation goes down, it reverts to frequently asked questions and agent transfer. The NIST AI RMF states that mechanisms should be in place to replace, deactivate, or shut down AI systems that demonstrate performance or outcomes outside their intended purpose, with roles and responsibilities assigned (MANAGE 2.4). These mechanisms must be built and tested prior to release.

Response table by failure type

Failure responses are organized into a single table. If any cell is left blank, no one knows what will happen when that failure occurs.

Human intervention points

Methods of human intervention can be divided into four categories.

There are three selection criteria: whether errors are reversible, whether throughput falls within review capacity, and whether risk is concentrated in a specific subset (restricted sales items, large monetary amounts, safety-related items). Anthropic explains that agents can pause for human feedback at inspection points or blocking points. The planner's job is to define those points in advance.

The experience of users waiting for cases routed to humans must also be designed. Set target review times, display progress status to users in the interim, and define how to handle cases that exceed the target time (assigning additional reviewers, safe fallback processing). Without processing rules for days when review queues overflow, human intervention directly translates into service delays.

It is common to blend multiple methods. For a classification workflow processing tens of thousands of cases per day, cases triggering risk conditions are handled via conditional escalation, the remainder through automatic application and post-hoc spot checks, and sellers dissatisfied with the outcome are provided an appeal path.

Reviewer authority and rationale

Having a human in the review loop does not mean oversight is functioning. If the review screen displays only the AI recommendation and a "Confirm" button, the reviewer can neither modify the recommendation nor inspect why that recommendation was generated. In form it is a human decision, but in substance it is automation.

For review to function substantively, three elements are required:

1. Decision rationale. The reviewer can view the primary evidence from which the AI produced the output (cited documents, factored items).
2. Authority. The reviewer can modify or reject the output, and that decision is actually reflected.
3. Records. The reasons for modifications and rejections are preserved to serve as material for future audits and improvements.

The NIST AI RMF states that human oversight processes should be defined, assessed, and documented in accordance with organizational policies (MAP 3.5). The AI Basic Act (Framework Act on the Development of Artificial Intelligence and Creation of Reliability Foundation) also includes, among the measures required of AI business entities (developers and deployers) providing high-impact artificial intelligence—that is, AI capable of significantly affecting life, safety, or fundamental rights in statutorily prescribed areas such as hiring and credit evaluation (see Chapter 1)—or products and services using it, human management and oversight, explanation measures, and the preparation and retention of documents verifying the measures taken (Article 34, Paragraph 1). Even for general customer service bots not classified as high-impact, substantive review design is fundamental to maintaining quality and trust.

Trust calibration and confidence display

It is problematic whether users trust AI outputs unconditionally or doubt them unconditionally. The Google PAIR Guide explains that the goal is for users to calibrate their trust appropriately so they trust the system in certain situations and double-check when necessary.

Displaying confidence levels can sometimes be helpful, but the method matters. The same guide warns that numeric confidence scores are risky because they assume users understand probabilities well; instead, it recommends presenting multiple candidates or categorizing confidence levels into discrete tiers, making clear what the user should do at each tier. If displaying confidence does not influence user decisions, not showing it at all is also an option.

User notices

Article 31 of the AI Basic Act imposes three requirements: when intending to provide products or services that utilize high-impact or generative artificial intelligence, the entity must notify users in advance of the fact that the product or service operates based on such artificial intelligence (Paragraph 1); when providing generative artificial intelligence or products and services utilizing it, the entity must label the fact that the output was generated by generative artificial intelligence (Paragraph 2); and when providing virtual audio, images, or video that are difficult to distinguish from reality, the entity must notify or label it in a manner that users can clearly perceive (Paragraph 3). Specific methods and exceptions are delegated to Presidential Decree (Paragraph 4).

The critical takeaway for planners is that prior notice and output labeling are, in principle, separate obligations. Providing prior notice does not exempt one from the obligation to label outputs. Specific methods and exceptions are stipulated in Article 23 of the Enforcement Decree. Prior notice can be given through inclusion in terms of service, on-screen display, posting at the place of provision, and similar means. Output labeling can be conducted through human-perceptible or machine-readable methods; if a machine-readable method is used, the fact that it was created by generative AI must be communicated at least once via text or voice. In cases where AI utilization is evident from the service name or on-screen copy, or where the system is used solely for internal business purposes, all or part of the provisions may not apply. Apart from legal obligations, the PAIR Guide recommends being upfront about what the product can and cannot do during initial use, and explaining the benefits to the user rather than the technology. Dissatisfaction arising from users misunderstanding the feature's scope (such as assuming they are viewing real-time information) is resolved by aligning expectations, not by modifying outputs.

Feedback and control

User feedback comes in two forms. As defined by the PAIR Guide, implicit feedback is user behavioral data captured in product logs (edits, regenerations, abandonment), while explicit feedback consists of opinions intentionally left by users (thumbs up/thumbs down, reports). If only a portion of the output is frequently incorrect, allowing users to edit the output directly and logging the revisions addresses both user burden and the need for improvement data. However, since user edits are not always correct, revision logs should be reviewed before being incorporated into evaluation sets or improvement data.

Control mechanisms are also necessary. The PAIR Guide recommends allowing users to customize or turn off outputs to fit their needs, offering options to try out or opt out of AI when it is first introduced, and respecting decisions not to use it. OpenAI advises providing easy ways for users to report inappropriate behavior, with humans monitoring those reports and responding appropriately. The NIST AI RMF likewise recommends establishing feedback processes through which end users and impacted communities can report issues and appeal outcomes (MEASURE 3.3).

Going deeper

The trap of perfunctory approval

A structure where a human approves every result may appear safe, but when throughput is high and the AI is mostly correct, reviewers increasingly approve without verification. If someone looks at hundreds of cases a day for a few seconds each and approves more than 99% as-is, that review is not oversight but merely a signature.

This trap is revealed in numbers. Regularly examine the reviewer's modification rate (the proportion of cases not approved as-is), review time per case, and whether the cases modified by the reviewer were actually incorrect. The tendency of humans to accept an AI's recommendations without scrutiny is called automation bias. If the modification rate is near zero and the review time is short, it is one of two things: either the AI is so accurate that prior approval is excessive, or the reviewer lacks the evidence and authority to make a determination and has given up on verifying. The former case is reduced through conditional escalation, while the latter case is resolved by redesigning the review screen. Another method is deliberately injecting known errors into some of the results to see if reviewers catch them.

The trap of automatic switching to another model

"Automatic switching to another provider's model" is frequently proposed as an incident response. Looking only at availability, it seems plausible, but there are three things a planner must verify. First, have you run the evaluation set on that model to confirm that it meets the quality threshold? Even with the same prompt, the output changes when the model changes. Second, do you have a contract and processing terms that permit sending customer data to that provider? Third, are latency and cost within contractual terms?

If even one of the three has not been verified, automatic switching merely turns an outage into a quality incident or a breach of contract. In such cases, graceful degradation or a queue is a safer answer. Maintaining a fallback model presupposes evaluating it alongside the primary model during normal operations and having a contract already in place.

A disclaimer is not a response

The phrasing "AI responses may be inaccurate" is common, but on its own, it is not a failure response. Users still believe the answer and act on it even after reading the disclaimer, and the disclaimer does not reduce fabricated answers by a single one. The notice required by law (the fact that it is AI-based, an indication that it is generated output) and failure handling (citing evidence, human intervention, fallback pathways) operate on different levels.

Notice is a matter of aligning user expectations, while failure handling is a matter of mitigating the harm of failures. If "Add a disclaimer" is entered in the failure response field of a planning document, that field is still empty.

At work

Great Light Lens's prescription guidance bot

The eyewear shopping mall Great Light Lens operated an AI bot that answered customer inquiries. Looking at the pilot operation logs, failures appeared in three types: a fabricated answer that advised "three free exchanges," which was not in the lens exchange policy; an over-refusal that rejected simple questions such as "How do I enter my prescription?"; and a formatting error where order lookup results appeared empty on the app screen.

The planner divided responses by type. For fabricated answers, the bot was made to respond by quoting sentences from the exchange policy document, and if it could not find a citation, to display "Cannot be confirmed in the policy" alongside a link to connect with an agent. For over-refusals, 120 rejected questions were added to the evaluation set to revise the prompt, the refusal rate was remeasured using held-out questions not seen during revision, and a direct shortcut to the prescription input screen was displayed whenever a refusal occurred. For formatting errors, order lookup results were fixed to a schema, and any request that failed validation was redirected to the default order screen.

There was one more thing. A feature was proposed where, when a customer uploaded a photo of a prescription, the AI would read the prescription values and enter them into the order. If the prescription was misread, an incorrect lens would be manufactured, and manufactured lenses were irreversible. The throughput was 60 cases per day. The planner chose a prior-approval method where the customer verified the AI-read prescription side by side with the prescription photo, and an optician approved every case before manufacturing. The discarded proposal was "automatic manufacturing for high-confidence cases." This was because with 60 cases per day, an optician could inspect all of them, and an incorrect lens could not be undone even if confidence had been high.

Gathered Beads Workshop's product inspection

The handmade crafts marketplace Gathered Beads Workshop sought to inspect 8,000 new product registrations per day using AI to filter out prohibited items (counterfeits, hazardous materials). The inspection staff could review 250 cases per day.

The initial proposal was "human review of all cases flagged as suspicious by the AI." Upon testing, cases flagged as suspicious by the AI numbered 900 per day. The planner divided the risks. Suspected hazardous materials were rare, but because the harm was severe, all of them were sent to humans. For suspected counterfeits, registrations with high confidence were placed on hold automatically, and only those with low confidence were sent to humans. Products without suspicion were passed automatically, followed by a daily audit of a 50-case sample. This reduced the cases routed to humans to 220 per day—combining 30 suspected hazardous materials cases, 140 low-confidence suspected counterfeit cases, and 50 post-audit sample cases—fitting within the review capacity of 250 cases. For sellers whose registrations were placed on automatic hold, an appeal channel was established where they could view the reason and request a re-review.

The review screen was redesigned as well. The initial screen contained only a "Suspected Prohibited" indicator and an approve button. Reviewers did not know why an item was flagged, so they approved most cases as-is, and the modification rate in the first week was 2%. The planner had the screen also display the product name, image area, and description sentences that the AI used as evidence, and required reviewers to select a reason if they changed a determination. As reviewers began actually correcting erroneous determinations after inspecting the evidence, the modification rate rose to 11% in the second week, and the recorded reasons served as material for the next evaluation set.

Common mistakes

1. "Adding the instruction 'Respond that you don't know' eliminates fabricated answers." It merely reduces them significantly; it does not eliminate them. Critical information requires a verification mechanism.
2. "If the answer remains the same across multiple generations, it is a fact." A model can repeat the same misconception. Consistency is not evidence.
3. "Refusal is safe, so it is not a failure." Over-refusal is a failure that blocks users and shifts costs to other channels.
4. "If structured outputs are used, schema validation is unnecessary." The schema can break during refusals or when hitting length limits.
5. "If an outage occurs, simply switch automatically to another model." Switching to a model that has not been evaluated or lacks data processing terms results in quality incidents or contractual breaches.
6. "Having a human approve every result is the safest approach." When throughput exceeds review capacity, it turns into perfunctory approval, and it imposes an excessive cost for easily reversible errors.
7. "If reviewers exist, oversight is functioning." Without evidence, the authority to modify or reject, and logging, oversight amounts to nothing more than a signature.
8. "Attaching a disclaimer constitutes failure handling." Notice is about aligning expectations; mitigating the harm of failure is a matter of design.

Summary at a glance

• Four failure types: fabricated answers (user is unaware), refusal and over-refusal (blocked), formatting errors (broken), and transient outages (waiting).
• Fabricated answers: allow "don't know," require direct citations, and verify citations. These only reduce errors; they do not eliminate them.
• Formatting errors: structured outputs + validation + reprocessing failed cases. Refusals and length limits are exceptions.
• Outages: bounded backoff retries if brief, queues if real-time processing is unnecessary, graceful degradation if prolonged, and fallback models presuppose prior evaluation and contracts.
• Human intervention: select prior approval, conditional escalation, post-audit, or appeals based on reversibility, throughput, and risk subsets.
• For review to function: evidence, authority, and records. Monitor perfunctory approval using modification rates and review times.
• Confidence displays should use tiers and action guidance; handle numerical confidence scores with caution.
• Article 31 of the AI Basic Act: Prior notice and output labeling are separate obligations. Reporting channels must be reviewed and answered by humans.

Check the official docs

• Claude Docs — Reduce hallucinations: Basic and advanced techniques, and the caveat that they "cannot be eliminated."
• Claude Docs — Structured outputs: Schema guarantees, and exceptions for refusals and max_tokens.
• Claude Docs — Claude API errors: Error codes, exponential backoff, and automatic retries in the SDK.
• OpenAI — Safety best practices: Human review, red-teaming, reporting channels, and human monitoring.
• Google PAIR Guidebook — Errors + Graceful Failure: Error types, next steps after failure, and returning control.
• Google PAIR Guidebook — Explainability + Trust: Trust calibration and methods for displaying confidence.
• Google PAIR Guidebook — Feedback + Control: Implicit and explicit feedback, editing, and turning off.
• NIST AI RMF — Core: MAP 3.5 (human oversight), MEASURE 3.3 (reporting and appeals), MANAGE 2.4 (replacement, decommissioning, and deactivation).
• Korea Law Information Center — Enforcement Decree of the AI Basic Act, Article 23: Methods of prior notice and output labeling, exceptions to application.

Self-check

4-1. State the three primary strategies that the Claude documentation cites for reducing fabricated answers.

4-2. Explain why over-refusal is not revealed by the error rate (proportion of incorrect answers), and state the two directions for addressing over-refusal.

4-3. Explain why a validation step must be retained in a feature using structured outputs even when the response completes normally without errors.

4-4. In an automated official document dispatch feature, the model API has not responded for 30 minutes. The dispatch only needs to be completed by tomorrow morning. Among retries, a queue, and graceful degradation, which is appropriate, and why?

4-5. State the four methods of human intervention, and write the condition suited to each in a single line.

4-6. A reviewer examines 500 cases per day for an average of 3 seconds each and approves 99.8% as-is. State the two situations this figure could signify, along with the corresponding action for each.

4-7. State what Article 31, Paragraphs 1 and 2 of the AI Basic Act respectively require, and explain the relationship between the two.

4-8. (Design) Write a five-row response table by failure type (prevention, detection, behavior upon failure, visible to user) for the AI feature you are currently responsible for.

Hands-on tasks