AI Audit Course

Contents

1. How to use this book
2. Chapter 1. Audit Planning, Scoping, and AI Inventory

   Questions answered by AI audits · What is considered AI · Roles and elements of an AI inventory · How to test the completeness of an inventory · Shadow AI · Risk-based scoping · Determining audit depth · Meaning and sources of audit criteria · Regulatory mapping 1: AI Basic Act · Regulatory mapping 2: Personal Information Protection Act (PIPA) · Framework mapping 3: NIST AI RMF, GAO Framework, ISO/IEC 42001 · Assurance, advisory, and objectivity · Using specialists and prior results · Whether to include embedded AI in the inventory · The boundary between audit criteria and legal judgment · Assurance after advisory: when the same team reviews twice

3. Chapter 2. Control Design and Operating Effectiveness

   Types of controls · Design effectiveness and operating effectiveness · Design deficiencies and operating deficiencies · Testing techniques and strength of evidence · When encountering deviations · Requirements for human review controls · Formal approval and automation bias · Selecting tests to identify formal reviews · Access control · Change management controls · Automated controls and benchmarking strategy · Vendor management controls · Reading service organization reports · The cost of meaningful human review · Agent tool permissions and human approval

4. Chapter 3. Evidence Collection, Testing, and Sampling

   Sufficiency and Appropriateness · Relevance, Validity, and Reliability · General Principles of Reliability · Procedures for Obtaining Evidence · Information Produced by the Company · Three Methods of Selecting Items · Sampling Methods · Factors Determining Sample Size · Evaluating Sample Results · Designing Logs Backward from Audit Questions · Reproducibility and Reperformance · Evidence Produced by the Audited Organization's AI · AI Tools Used by Auditors · Why sampling is still necessary in an era of full-population analysis · Integrity of agent logs

5. Chapter 4. Model and Data Risk Verification

   Validity, reliability, accuracy, and robustness · Confusion matrix and metrics · Baselines · Criteria required of an evaluation set · Measuring the quality of generative AI · Three categories of bias · Subgroup performance verification · Data provenance and lineage · Label quality · Operational shifts and monitoring · Change types and scope of reverification · Reverification procedures and approval criteria · There is no single standard for fairness · Performance when humans and AI work together

6. Chapter 5. Reporting and Follow-up

   The four elements of a finding · Identifying causes · Estimating and expressing effect · Risk rating and prioritization · Writing recommendations · Obtaining management comments · Expressing conclusions and assurance · Disclosing scope, methodology, and limitations · Principles of follow-up verification · Closing evidence by action type · Risk acceptance and reporting · The impact of AI findings reaches people · When opinions differ with management

7. Appendix A. Glossary
8. Appendix B. Self-check answers and explanations
9. Appendix C. Exam day guide
10. Appendix D. Reference documents