AI Security Course

Contents

1. How to use this book
2. Chapter 1. Threat modeling

   Why LLM apps need their own threat model · The four questions of threat modeling · Assets: what are you protecting? · Attack surface and attacker control · Trust boundaries and data flow diagrams · Applying STRIDE to LLM components · OWASP Top 10 for LLM Applications 2025 · Chains across categories: mapping a single incident across multiple OWASP categories · Extending to agents: OWASP Top 10 for Agentic Applications 2026 · Framework map: NIST AI RMF, Google SAIF and MITRE ATLAS · Prioritizing risks and recording residual risk · Stateful systems: entry and impact happen at different times · Turning a threat model into requirements and verification items

3. Chapter 2. Prompt injection

   Definition and root cause · Direct and indirect injection: two threat models · How it differs from jailbreaking · The five stages of a successful attack · Path map: everything the model reads on the user's behalf · Tool results and tool definitions are inputs too · Three branches of damage: action hijacking, exfiltration and answer contamination · Why there is no perfect block · Mitigation 1: separation, labeling and structuring · Mitigation 2: screening and detection · Mitigation 3: privilege separation, confirmation, output restrictions and their combination · The boundary between legitimate work and attacks: quotations, translation and security training · Injection across multiple turns and memory · The trap in human confirmation: approval fatigue and trust exploitation

4. Chapter 3. Data protection and permissions

   Where sensitive information leaks: the data lifecycle · Minimization, redaction and pseudonymization · Terms for data sent to external AI services · System prompts are not secrets · Permissions in RAG and vector stores · Excessive agency: three roots and eight preventive measures · Turning least-privilege tools into a design · Delegation and identity: whose authority is used? · Output is untrusted input · Rendering output safely · Model-chosen addresses and code: SSRF and code execution · Binding approval to execution · Isolating derived data: caches, memory, summaries and exports

5. Chapter 4. Defense testing and red teaming

   Guardrail map: what goes where? · When does checking finish? The Agents SDK example · Classifiers: types of decision engine and their limitations · Failure behavior: when no judgment is returned · Definition of red teaming and rules of engagement · Designing attack scenarios · Broadening scope with existing attack knowledge bases · Building the evaluation set · Judging success: use outcomes · Metrics for defense effectiveness · Continuous monitoring and regression · How many runs? Interpreting probabilistic results · Experiments comparing control combinations

6. Chapter 5. Operational security and incident response

   What must audit logs prove? · Protecting the logs themselves · User identification and abuse tracking · Forms of unbounded consumption · Budgets and blocking mechanisms · Supply chain 1: models and model files · Supply chain 2: tools, plugins and MCP servers · Configuration inventories, AI BOM and the model lifecycle · Designing detection signals · Incident response procedure · Containment switches and safely reduced operation · Personal-information breach notification and records · Balancing evidence preservation and privacy · Recovering poisoned memory, indexes and caches

7. Appendix A. Glossary
8. Appendix B. Self-check answers and explanations
9. Appendix C. Exam day guide
10. Appendix D. Reference documents