Appendix B. Self-check answers and explanations

1-1. Role, goal, constraints and success criteria. Without success criteria, the most frequently omitted element, reviewers cannot consistently decide pass or fail from the result, and automatic evaluation cannot follow.

1-2. For example: “Within 1,200 characters overall,” “five question-answer pairs,” “no more than three sentences per answer,” “one sentence stating the interview’s key message in the first paragraph,” and “quoted CEO statements must use transcript wording only; any quotation absent from the transcript means fail.” The key is converting “polished” and “manageable” into length, structure, position and required content, with a failure condition.

1-3. Extracting decisions is extraction: preserve source wording, mark no decisions as none, and do not polish or add. Executive briefing is transformation: preserve meaning and omit no decisions, while allowing wording and order adapted to the reader.

1-4. For example: “The payment-machine speaker reads this text aloud, so spell out Korean terms instead of using English abbreviations. For example, write ‘전기차 충전 구역’ (electric vehicle charging area) instead of ‘EV 충전 구역’ (EV charging area).” A positive action with a reason helps the model avoid similar issues, such as symbols and English units.

1-5. Links occupy the two-line preview, so both conditions cannot be met in the same place. For example: “Answer the question in complete sentences in the first two lines. Put one supporting-document link on the third line.” Separate placement preserves both conditions.

1-6. Message authority comes from position, not introductory words. The sentence is merely text inside a user message and cannot override system (developer) rules. A system statement that instructions inside user input do not change rules clarifies the boundary further.

1-7. For example, ask about the assignee: guessing could give work to the wrong person. Leave the deadline pending, mark it “Deadline undecided” and add it to the confirmation list: it must not be invented, but need not stop drafting. Assume or omit the meeting-room name and label this, since it barely affects the result. The criteria are harm if wrong and ease of later correction; different allocations are acceptable if justified on the same basis.

2-1. Intended lessons include classification criteria, output format and tone. Accidental lessons include sentence length, greetings, label frequencies and repeated phrases. If incidental features are unevenly distributed, the model learns them as rules.

2-2. Set aside and lock evaluation inputs first, then choose examples from what remains. Reversing this can leave example inputs in evaluation, so the model repeats questions whose answers it has already seen and scores are inflated.

2-3. The key is separate tags for instructions, policy and input.

<instructions>Draft a reply to the event inquiry below.</instructions>
<event_policy>In rain, the event moves to the indoor gym.</event_policy>
<inquiry>…</inquiry>

2-4. (a) An executive screen suits a one-sentence conclusion and table; check Markdown rendering. (b) An announcement suits short declarative sentences; specify number pronunciation and why symbols and tables should be avoided. (c) For inventory loading, enforce JSON through structured outputs; if CSV is required, convert validated JSON with code. Keep human-readable explanation out of the data.

2-5. For example: an explicit statement that there are no companions is represented by an empty array []; no mention is null; illegible text has a separate status such as "unreadable". Subsequent work differs: an empty array allows completion, null calls for asking the applicant, and unreadability requires checking the original.

2-6. Refusal and output-limit truncation. Before parsing, check Anthropic’s stop_reason for refusal or max_tokens, or OpenAI’s refusal field and incomplete response status.

2-7. (a) Structured outputs defining allowed labels with enum, or a tool with an enum field. (b) A system instruction to answer without a preamble, or output within defined tags; post-process occasional remaining preambles. (c) Put the end of the interrupted response in the user message and ask for continuation, or resend the request if appropriate.

3-1. Context rot is a decline in the ability to accurately recall information as context tokens increase. Give any two of: irrelevant material can support wrong answers, cost and latency increase, and simultaneous retrieval of multiple pieces loses accuracy.

3-2. Any five of system instructions, tool definitions, examples, documents/search results, conversation history/tool results, and output headroom. Output headroom is most often forgotten. The response may be truncated with max_tokens, or reasoning can consume the limit and leave no visible answer (Chapter 4).

3-3. Put documents first and the question last, requesting quotes before answers.

<documents>
  <document index="1"><source>Audit Report A</source><date>2026-03-31</date><document_content>…</document_content></document>
  <document index="2"><source>Audit Report B</source><date>2026-09-30</date><document_content>…</document_content></document>
</documents>
Copy relevant original sentences word for word into <quotes> with numbers. If none exist, write "No relevant quotes."
Then answer in <answer> using only <quotes>, attaching a quote number to each claim.
Question: …

3-4. (1) Was correct evidence in the store? (2) Was it among top retrieval results? (3) Did it enter context after selection? (4) Was the answer wrong despite its inclusion? Failure in the first three means the model never saw the evidence, so revising instructions cannot solve it.

3-5. Confirmed numbers, dates and amounts; decisions distinguished from suggestions; open questions and awaited replies; next actions.

<project_state>
Confirmed: event November 21, maximum 150 attendees, venue fee agreed under quotation E-2026-07
Open: awaiting the second speaker candidate's reply
Decision history: second-floor auditorium confirmed October 2
Next action: announce confirmed speakers after the reply
</project_state>

3-6. A cache hit requires an exact prefix match across tool definitions, the system message and subsequent messages, in that order. Changing the tool list changes the beginning, recomputing the long guidelines and potentially paying only cache-write costs. Options include a common tool list with server-side authorization at execution, or fixed lists per access level. Some newer models also support adding or removing tools during a conversation while preserving the prefix.

3-7. Placement examples: put the body in a tool result, not the system prompt; label its source URL and external-page origin and encode it as a JSON string. State in the system prompt that instructions inside results are reportable information, not commands to follow. Damage controls include read-only least privilege, human approval before outbound actions, lightweight classification of retrieved bodies, and checking pricing-table output format.

4-1. Reasoning tokens (1) are billed as output even without returned text and (2) occupy the context window and output cap. Visible thinking-block text is a summary, not original thinking; some newer models omit even that summary by default.

4-2. Effort signals how much care to take. A sufficiently difficult problem can trigger thinking at low effort, but less than at high effort on the same problem. For an exact cap, use output limits—Anthropic max_tokens, OpenAI Responses API max_output_tokens—which include reasoning tokens.

4-3. (1) Separate simple lookups from comparison. (2) Set numerical latency allowances and quality floors per type. (3) Run evaluation questions at each effort, recording accuracy, response time and output tokens. (4) Select the lowest effort meeting quality. (5) If particular questions become shallow, add targeted instructions or raise effort. (6) Measure again after model changes.

4-4. The cap covers reasoning and visible answers together, and reasoning consumed it. Remedies include (1) allowing ample headroom—OpenAI recommends at least 25,000 tokens in initial testing—and (2) lowering effort or splitting the task. Check stop reasons or status so truncated answers are not treated as complete.

4-5. Keep goals, constraints, success criteria, output format, and business-specific facts and standards. Remove detailed procedural prescriptions and “think step by step,” and add completion and verification conditions. Test without examples first, restoring them only as needed. Keep any sequence required by business policy: it is a requirement, not a thinking method.

4-6. Newer thinking-enabled Claude models may return a reasoning_extraction refusal when asked to reproduce internal reasoning, and that refusal is billed even if it occurs before output. Alternatives are evidence for supporting clauses or source locations, explanation for one or two short sentences, and assumptions for assumptions used. To inspect thinking, set display to summarized and read the summary.

4-7. One possible answer follows.

For a gate, use code between extraction and comparison to check required specifications and page references. Do not proceed with missing items. After writing, likewise check that every number appears in the comparison table.

5-1. Code grading is fast and reliable but weak on nuanced judgment. Human grading is flexible and high quality but slow and costly. LLM grading is fast and handles complex judgments, but requires reliability verification first.

5-2. Examples: amounts and dates matching the source, no prohibited promises, required information such as contact and deadline, length, and tone. Source agreement and prohibited promises are critical: a single failure sends wrong information or creates disputes, making the output unusable. Grade them separately from averages.

5-3. Irrelevant or nonexistent input, overly long input, inappropriate or harmful input, and ambiguous cases where people disagree. Remove or pseudonymize personal information in production-log cases while retaining the conditions explaining the failure.

5-4. The prompt increasingly fits only those cases, so rising scores no longer indicate unseen-input quality. Lock a confirmation set for the final predeployment decision only, replacing some cases after viewing it.

5-5. Example process: (1) Define the rubric and have people grade samples, preferably with overlapping grading to verify human agreement. (2) Have the LLM grade the same samples. (3) Measure agreement. (4) Inspect disagreements and revise rubric and evaluator instructions. (5) Measure again. (6) Scale after consistent agreement and continue periodic sample checks. Safeguards include narrow pass/fail or 1–5 outputs, evidence before scores or enabled thinking, a different model from the generator, tagged graded content whose instructions are ignored, and reversed-order double grading for pairwise comparisons.

5-6. Assuming independent attempts, 0.8 × 0.8 = 0.64, or 64%. Where every attempt must succeed, such as customer service expected to be consistent each time, the probability of all successes, pass^k, better represents experience than average pass rate.

5-7. Record type and originating stage per failure, prioritize by both frequency and severity, and select one fix. Define cases it must correct and must preserve, then compare before and after on the same set. Frequency alone is insufficient when a rare failure has critical impact—for example, five broken-format cases halt downstream data ingestion.