Appendix B. Self-check answers and explanations

1-1. Agents are suitable for open-ended problems where it is difficult or impossible to predict the number of required steps in advance. The trade-off is increased costs and the potential accumulation of errors resulting from autonomy, and broad tool permissions must also be taken into account. Anthropic therefore recommends sufficient testing in a sandbox and appropriate guardrails. If the steps are fixed, a workflow is the simpler answer.

1-2. The guide cites six cases: tasks where predictability must be maintained, tasks that provide static or constrained information, tasks where costly errors must be minimized, tasks that require complete transparency, tasks where building quickly and cheaply is most important, and automating tasks that people value and want to do themselves. In these tasks, rules, search, and human judgment are likely to be better than AI.

1-3. Rules and lookups are appropriate. Because the number of remaining seats already exists as an exact value in the reservation system, you only need to lightly understand the intent of the question and display the retrieved value as is. If a generative model turns numbers into sentences, there is nothing to gain, and it only introduces costs and the risk of misrepresenting numbers.

1-4. Start by creating business procedures and storage structures to record visit history in the system. A predictive model can only be built when historical records exist, and can only be evaluated when expected outcomes exist. If only paper registries exist, even if development begins, there is no way to verify the results.

1-5. Article 34, Paragraph 1 of the AI Basic Act requires establishing and operating risk management plans, establishing and implementing explanation plans to the extent technologically feasible, establishing and operating user protection plans, human supervision and oversight, and preparing and retaining documentation to verify measures taken to secure safety and reliability. Separately from this, prior notice under Article 31, Paragraph 1 applies, and Article 35, Paragraph 1 stipulates that efforts should be made to conduct a high-impact artificial intelligence impact assessment that evaluates the impact on human fundamental rights in advance.

1-6. Under the proviso of Article 37-2, Paragraph 1, the right to refuse may be restricted if the decision is based on consent, statutory provisions, contract performance, or the like. However, the data subject's right to request an explanation and review of the decision (Paragraph 2), the controller's resulting obligation to take measures (Paragraph 3), and the obligation to publicly disclose the criteria, procedures, and personal information processing methods of the decision in an easily accessible manner (Paragraph 4) still remain. According to the Enforcement Decree, unless there is justifiable cause, the controller must provide a concise and meaningful explanation containing the results, key criteria, and processing procedures of the decision, or review and notify whether the opinions submitted by the data subject were reflected. A restriction on the right to refuse does not mean an exemption from the entire article.

1-7. The answer varies depending on the task. A good response answers the five questions in sequence (Can it be determined by rules? Is showing the original text sufficient? Do the benefits outweigh the costs? Are there records and expected outputs? Does an error lead to irreversible damage?) and reaches one of the six conclusions (Proceed (automation), Proceed (assistive format), Change form (rules/search), Data prerequisite, Hold (maintain current method), Do not build) based on the first question triggered. If the conclusion is to proceed, preparing an evaluation set is recorded as the next action.

2-1. The four elements that must be included are what (metric), where (data), how much (threshold value), and how (measurement method). If even one is missing, others cannot determine pass or fail. If a baseline (what figure it is being compared against) is also included, you can explain why the threshold value is that specific number.

2-2. For example, write it using percentiles and time limits, such as "In 95% of production requests, the first token appears within 1.5 seconds, and the total response completes within 8 seconds." For features where users wait in front of a screen, use time to first token (TTFT), which governs perceived latency, as the primary metric, and total time as a secondary metric. The measurement method must also specify where it is measured (whether based on the user's device including the network, or based on the server's API response) so that decisions can be judged against the same criteria as the development team.

2-3. Precision is the proportion of actual positives among those predicted as positive, and recall is the proportion captured as positive among actual positives. The former measures "how many of what was caught are correct," while the latter measures "how much of what should have been caught was caught."

2-4. Precision must be maintained. Because flagged cases immediately lead to appointment cancellations without human confirmation, mistakenly capturing a text message that is not a cancellation request causes an incident where a valid appointment is deleted. A missed cancellation request has relatively low cost if a staff member can re-read the text or the patient calls back to handle it.

2-5. When the model refuses for safety reasons (when stop_reason is refusal), the refusal message takes precedence over schema constraints, so the output may not conform to the schema. When the output is truncated upon reaching the max_tokens limit, the result may also be incomplete and fail to conform to the schema. Therefore, validation and failure-handling rules are required.

2-6. (A) The rate of remaining account numbers leads to a personal information incident, so it is a launch blocker. (B) The conciseness score is a quality item that does not impede usage, so it is an improvement target. (C) The rate of incorrect refund deadlines is a launch blocker because customers could act on the incorrect deadline and suffer damages.

2-7. A good response specifically details the copy to show the user and the next action (manual entry, relevant menu, connecting to a representative, when to retry) for each failure scenario (unable to answer, out-of-scope input, insufficient evidence, processing delay, system outage, etc.). If there is a field without a next action, such as "An error occurred," the design is not yet complete.

3-1. The sequence is: defining evaluation goals, collecting data, defining evaluation metrics, running and comparing, and continuous evaluation. Because the results of a preceding step are the inputs to the next step, the order cannot be changed.

3-2. Irrelevant or non-existent inputs, excessively long inputs, low-quality, harmful, or irrelevant user inputs in the case of chat features, and ambiguous edge cases where even humans disagree.

3-3. Code-based grading lacks nuance for complex judgments that are difficult to express in rules, and struggles with correct answers that differ from the ground truth only in expression. LLM grading can produce slight variations in results, so its reliability must be verified first and calibrated against human grading. Human grading has high quality, but is slow and expensive.

3-4. The return due date is graded using code to check whether it exactly matches the value in the loan system. Because the correct answer is fixed, it is the fastest and most reliable. Because instructional tone has no fixed ground-truth string, grade it with an LLM grader supplied with a rubric, but only use it after first comparing it with human-graded samples to confirm agreement.

3-5. Capability evaluation observes how well the system has learned to perform tasks it previously could not do well, starting from a low pass rate. Regression evaluation checks whether it continues to perform well on tasks it already did well, and the pass rate must be nearly 100%. When capability tasks are consistently performed well, they graduate to the regression suite and continue to run to monitor for performance degradation.

3-6. Overfitting resulting from tuning specifically to the development set should be suspected. While there was a 23 percentage point increase on the development set, it only increased by 2 percentage points on the holdout set that was unseen during tuning, indicating that the actual improvement is small. Base launch decisions on the holdout set results, and examine cases that were fixed only in the development set to identify the root cause.

3-7. For a feature where a human selects one among multiple candidates, pass@k is appropriate because it only needs to succeed at least once out of k attempts. For a feature where the customer must receive the same quality answer every time, pass^k is appropriate because all k attempts must succeed.

3-8. A good response completes all fields of the template in this chapter, and for each success criterion, specifies the grading method (code grading if it can be checked against ground truth, LLM grading calibrated against humans if it can be evaluated with a rubric, or human grading if expert judgment is required) along with the rationale. The evaluation set field must separately specify the composition relative to real-world proportions, an edge-case suite, a high-risk suite, and the holdout set reserved from each suite.

4-1. Permitting the model to say "I do not know," extracting direct quotes from long documents first to serve as grounding, and verifying each claim against citations while withdrawing claims for which grounding could not be found.

4-2. Over-refusal does not provide incorrect information, so it is not captured in the incorrect answer rate (error rate). If accuracy is measured across all questions, refusals are counted as non-correct answers and lower the score, but unless the refusal rate is viewed separately, it is not apparent whether that drop was caused by refusals. There are two directions to address this: adding answerable refusal cases to the evaluation set and fixing them to reduce refusals themselves, and presenting next actions such as relevant menus or connecting to a representative even when refusing.

4-3. According to Claude documentation, structured outputs do not guarantee the case of string enum values (enum, const), meaning that even a response that terminated normally without errors or special stop reasons may contain a value differing only in case. Therefore, keep validation that compares enum values case-insensitively. Because outputs can also deviate from the schema due to refusals (stop_reason is refusal) or reaching max_tokens, combine stop reason checks with schema validation.

4-4. A queue is the right choice. For an outage that has persisted for 30 minutes, a few retries are not appropriate; sending does not require real-time execution, and there is ample time until tomorrow morning. Retain the requests and process them after recovery, notifying the person in charge of the estimated completion time. Graceful degradation is the appropriate response when providing a simpler path for a feature where the user needs to see results immediately.

4-5. Pre-approval is appropriate when outcomes are irreversible and throughput is within review capacity; conditional escalation is appropriate when throughput is high and risks are concentrated in a specific subset; post-hoc sample audits are appropriate when errors can be easily corrected and damages are minor; and appeal processing is appropriate when outcomes affect individuals and the affected party can notice the error.

4-6. First, it may be a situation where the AI is indeed highly accurate, making pre-approval for every case excessive. In this case, reduce it to conditional escalation, routing only cases that trigger risk conditions or low confidence. Second, it may be rubber-stamping where reviewers gave up on verifying because they lacked judgment rationale or editing permissions. In this case, provide rationale, modification and rejection capabilities, and reason logging on the review screen. Deliberately inserting known errors to see if reviewers catch them can distinguish between the two.

4-7. Paragraph 1 requires that when intending to provide products or services that utilize high-impact or generative artificial intelligence, users must be notified in advance that the product or service operates based on such artificial intelligence. Paragraph 2 requires that when providing generative artificial intelligence or products or services utilizing it, it must be indicated that the outputs were generated by generative artificial intelligence. The two are distinct obligations, so fulfilling one does not exempt fulfilling the other.

4-8. A good response sets at least fabricated answers (hallucinations), over-refusals, formatting errors, transient failures, and prolonged outages as rows, and fills each row with prevention (grounded citations, evaluating refusal cases, structured outputs, gradual traffic increases, preparing fallback paths), detection metrics, fallback behavior on failure, and user-facing copy and next actions. If there is a cell containing only a disclaimer, that cell does not count as a response measure.

5-1. It must be capable of exposing problems, representative, and attributable to changes.

5-2. Because shadow testing does not show new results to users, it cannot measure user reactions such as reports, churn, or conversions. Use canary releases if user reactions must be observed and the change is easy to roll back; use A/B experiments if comparing the business metric impact between two options.

5-3. Differences observed in small samples may be coincidental, making it impossible to determine whether a threshold was crossed. Without a minimum sample size, rollbacks occur due to random noise, or rollouts expand to subsequent stages without justification. A minimum sample size is a prerequisite for judgment.

5-4. The four layers are system (errors, latency, cost), quality (citation grounding rate, schema validation failures, refusal rate, sample grading), user behavior (regenerations, agent handoffs, drop-offs, reports), and business (conversions, cost savings). From the system's perspective, fabricated answers (hallucinations) are returned as normal responses, so nothing is captured in the error response rate.

5-5. Active (fully supported, recommended), Legacy (updates discontinued, may be deprecated in the future), Deprecated (functional but not recommended, with replacement models and retirement dates specified), and Retired (unavailable). Requests sent to retired models fail.

5-6. The sequence is: containing impact (turning off, routing to humans, rolling back), identifying affected individuals followed by notification and correction, fixing root causes and restoring service (gradually reopening after verification via regression evaluation), recording, blameless post-mortem, and implementing and verifying preventive measures.

5-7. The first type is 50 cases × KRW 40,000 × 10% exposed to customers = KRW 200,000 per month; the second type is 1,200 cases × KRW 2,000 = KRW 2,400,000 per month. In terms of monthly damage, the second type comes first. However, if the first type could lead to legal or safety issues or major trust damage, verify separately whether blocking mechanisms are sufficient.

5-8. A good response includes all of the following: rollout scope, rollout strategy and rationale for selection, step-by-step percentages, durations, and minimum sample sizes, control groups, quantitative abort criteria, rollback procedures (including already-processed results), operational metrics and alerts, change management, and incident escalation/contact systems. If abort criteria lack a minimum sample size or if the control group is a before-and-after comparison, revise it again.