AI Product Planning Course

Contents

1. How to use this book
2. Chapter 1. Evaluating AI Applicability

   What is evaluating AI applicability? · The ladder of solutions · When rules and lookups win · When search wins · Problems requiring models · Problems where AI is not appropriate · Automation and augmentation · Criteria for choosing between workflows and agents · Data conditions · Cost conditions · Applicability to high-impact areas · Automated decisions and the Personal Information Protection Act (PIPA) · Questions to filter out demands that do not require AI · AI applicability evaluation memo · The trap of deciding to "wait until the model gets better" · Does risk disappear if a human decides? · Items easily overlooked in cost comparisons

3. Chapter 2. Requirements and Success Criteria

   How requirements for AI features are different · Verifiable success criteria · Multiple dimensions of success criteria · Precision and recall · Procedure for turning vague requirements into criteria · Input contracts · Output contracts · Balancing quality, latency, and cost · User-facing failure experiences · Launch-blocking criteria and improvement targets · Rationale for threshold values · Structure of a requirements document · The temptation to reduce everything to a single metric · Error magnitude also serves as a criterion · When is consistency a requirement?

4. Chapter 3. Evaluation Design

   What is evaluation? · The five stages of the evaluation process · Representativeness of the evaluation set · Edge cases and rare high-risk cases · Sources and size of the evaluation set · Held-out sets and overfitting · Three grading methods · Order of selecting grading methods · Making LLM graders reliable · The role of human grading · Capability evaluations and regression evaluations · Offline evaluations and online metrics · Succeeding once versus succeeding every time · Deciding whether to launch based on evaluation results · Evaluation plan · Does a higher score mean it has improved? · The lifecycle of an evaluation set · The attribution problem of online metrics

5. Chapter 4. Failure Handling and Human Oversight

   Failure is part of planning · Fabricated answers · Refusal and over-refusal · Formatting errors · Transient disruptions and retries · Fallback paths and graceful degradation · Response table by failure type · Human intervention points · Reviewer authority and rationale · Trust calibration and confidence display · User notices · Feedback and control · The trap of perfunctory approval · The trap of automatic switching to another model · A disclaimer is not a response

6. Chapter 5. Launch and Operations

   Launch is the beginning of measurement · Defining launch scope · Comparing launch methods · Canary design · Abort criteria and rollback · Conditions for A/B testing · Tiers of operational metrics · Alert design · Model and prompt change management · Incident response · Blameless postmortem · Prioritizing improvements with operational data · Phased rollout plan · The temptation to change metrics · Silent quality degradation · The decision to turn off a feature

7. Appendix A. Glossary
8. Appendix B. Self-check answers and explanations
9. Appendix C. Exam day guide
10. Appendix D. Reference documents