Appendix C. Exam day guide
The day before the exam. Re-read "Summary at a glance" and "Common mistakes" in each chapter. In particular, check whether you can distinguish the following five pairs. Precision and recall, release-blocking criteria and improvement targets, capability evaluation and regression evaluation, shadow testing and canary, advance notice and output labeling.
Basic exam information. 35 questions, 60 minutes; 2-point (principle verification and application) and 3-point (analytical) questions are converted to a 100-point scale, and 75 points or higher is a pass.
Time allocation. Since you solve 35 questions in 60 minutes, it is an average of about 1 minute and 40 seconds per question. Solve all questions once within the first 40 minutes, mark any questions you are unsure about, and return to them in the remaining time. Because 3-point analytical questions have lengthy scenarios and many numbers, getting into the habit of transcribing the numbers into a table first saves time.
Order of reading the scenario.
- Find the bold conditions first. Throughput, reversibility, data availability, deadlines, human review capacity, and contract terms are mostly here.
- Check what the question asks you to select (implementation method, scoring method, release method, what to do right now).
- Before reading the options, come up with one line for "what would be correct under these conditions."
- For each option, check whether you can state "which condition makes it incorrect."
Same options, different scenarios. This exam uses the same list of options across different scenarios. Do not choose what appears to be the most responsible option just by looking at the options. "A human approves all results," "Rebuild the evaluation set with actual requests," and "Release in phases" can each be the correct answer or an incorrect answer depending on the scenario. The basis for the correct answer is always in the conditions within the scenario.
Tips by format.
| Format | Tips |
|---|---|
| Single choice | When two answer candidates remain, find which one violates one of the bold conditions |
| Multiple choice | Evaluate each option independently, and do not force a selection to match a count |
| Statement evaluation | Evaluate the conclusion and the reason separately. If the reason is incorrect, No |
| Sequence ordering | Write down what input each step uses, and arrange them in the order that inputs are produced first |
| Matching | Eliminate clear pairs first, and remember that there is one unused option |
Final checklist.
- In calculation questions, did you fail to account for "the proportion actually delivered to customers" or "cases already blocked"?
- Did you skip a high-impact review on the grounds that a human makes the final decision?
- Did you mistake consistency or disclaimers for evidence or a countermeasure?
- Did you choose expansion or rollback despite having an insufficient sample?
- In a pre-release scenario, did you select operational logs or A/B testing?
After the exam. For incorrect answers in practice sets, write down a single line along with the explanation: "Which condition did I miss?" If you missed the same condition twice, re-read "Common mistakes" in the chapter where that condition appears.