Chapter 5. Risk assessment and accountability

Learning objectives

• Write AI risks as harm scenarios identifying affected people, triggering conditions and harm.
• Explain NIST AI RMF's four functions and seven trustworthiness characteristics, and the Generative AI Profile's risk categories.
• Distinguish four risk response options and residual risk, and allocate controls proportionate to risk level.
• Design legally required records and records needed to explain organizational decisions.
• Explain the purposes and limits of management system certification and frontier safety frameworks.

Introduction

Suppose a company classifies all twenty AI use cases as high-risk, each requiring prior committee review, dual review and monthly reporting. Three months later, paperwork overwhelms the committee, preventing deep attention to the two cases actually affecting customers. Conversely, marking everything low-risk means no one looks. Risk assessment does not eliminate risk; it determines where limited attention and resources go. Trying to protect everything equally prevents adequate protection anywhere.

These decisions must also be explainable later. Whether a regulator, customer or next year's responsible officer asks, records answering "Why did we choose these controls, and what did we accept?" are the substance of accountability.

The frameworks in this chapter organize judgment; they are not legislation. Pay particular attention to their interaction with Chapters 2 and 3's legal obligations. We begin with writing risks, then examine AI RMF structure, trustworthiness characteristics and generative AI risk categories. Next come response choices, level-based control mapping, the distinction from legal requirements, records and audits, and operational monitoring. Finally, we consider what ISO/IEC 42001 and frontier safety frameworks do and do not tell us.

Key concepts

Writing risks as harm scenarios

Labels such as "bias risk" or "hallucination risk" cannot be assessed. To measure and control risk, write who suffers what harm under which conditions.

Poor example: Bias risk
Good example: An intent classifier mistakes voice inquiries containing dialect for "not a complaint",
        delaying older customers' cancellation requests beyond the average. (Affected people: older customers;
        condition: nonstandard speech; harm: delays and additional charges; recovery: partly possible through refunds)

For each scenario, assess impact (harm severity, effects on rights, recoverability and number affected) and likelihood (frequency and prevalence of conditions). Set levels around people and rights rather than cost. Warehouses and schools have different safety inspection cycles because of the people inside, not the building price.

NIST AI RMF structure and starting flow

The NIST AI Risk Management Framework (AI RMF 1.0, released 26 January 2023) divides AI risk management into four functions.

NIST explains that once Govern is established, most organizations begin with Map and proceed to Measure or Manage, moving among functions iteratively as needed. This is a cycle rather than a sequence completed once.

Seven characteristics of trustworthy AI

AI RMF lists seven characteristics: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair with harmful bias managed.

NIST describes "valid and reliable" as a necessary condition of trustworthiness, and "accountable and transparent" as related to all the other characteristics. Characteristics can conflict and must be balanced in context. More detailed explanations, for example, may conflict with privacy or security.

Three similar-sounding characteristics answer different questions. Transparency asks what happened in a system; explainability asks how a decision was reached; interpretability asks why it arose and what it means to the user. Reconstructing inputs and model versions through logs is transparency; helping customers understand the reasons for their outcomes is interpretability.

The Generative AI Profile's twelve risks

NIST AI 600-1 (Generative AI Profile, July 2024) organizes risks unique to or amplified by generative AI into twelve categories.

Use the table as a checklist to avoid missing harm scenarios. Review all twelve categories for each case, select those applicable, and write concrete scenarios for each selected category.

Four risk response options and residual risk

AI RMF prioritizes treatment of documented risks by impact, likelihood and available resources (MANAGE 1.2). Its response options are mitigating, transferring, avoiding and accepting (MANAGE 1.3).

Risk remaining after controls is residual risk. AI RMF calls for documenting negative residual risks to downstream purchasers and end users (MANAGE 1.4). Record residual risk rather than hide it, obtain the acceptance authority's signature, and inform recipients of the system when needed. Transfer does not eliminate risk either. Contracts can share costs, but affected people still suffer harm, so risks with major human impact must not be handled solely by transfer.

Mapping controls to risk levels

This chapter's practical skill is mapping controls to risk levels. Predefined control packages make decisions faster and more consistent.

Avoid two assumptions when mapping. First, the same level does not always require identical controls. Different causes require different responses: input minimization for information exposure, source verification and human review for incorrect answers. Second, a low score does not exempt legal obligations. That is the next section's topic.

Distinguishing legal requirements from discretionary controls

Risk register controls mix two different types: legally required measures and organizational choices proportionate to risk.

A "Basis" column marking law, contract or internal judgment lets you reduce discretionary controls proportionately without missing legal requirements.

Records and audits

Accountability is tested by whether you can explain decisions later. The following records are legally required or practically essential.

Version control is crucial. Overwriting risk management documents on every model change prevents explanation of past decisions. Keep each version with a date. Records themselves also involve personal information processing: logging entire prompts creates new risk. Record only necessary items, restrict access and define retention. Records retained under other laws must be kept separately from other personal information under PIPA Article 21(3).

Audits examine two things separately. Design adequacy asks whether controls are properly designed to address risk; operating effectiveness asks whether they actually worked. Rules may require human review while samples show no evidence of it.

Operational monitoring and incident response

AI changes after deployment: input distributions shift, vendors update models, and users find unexpected uses. AI RMF calls for implementing post-deployment monitoring plans (MANAGE 4.1), communicating incidents and errors to stakeholders including affected communities (MANAGE 4.3), and procedures for end users and affected persons to report problems and challenge outcomes (MEASURE 3.3).

A monitoring plan specifies three things: what to watch (error rates, group performance gaps, complaints, human-review correction rates); when to act (thresholds metrics must not cross); and who does what (suspension, reassessment, notification). Connect it to statutory notices too. Chapter 3's 72-hour procedure begins when conversation-log leakage becomes known or the Decree's possible-breach conditions apply, such as confirmed unlawful access where identifying the data subjects is difficult.

ISO/IEC 42001: management system certification

ISO/IEC 42001 (2023) is an international standard specifying requirements for establishing, implementing, maintaining and continually improving an AI management system in organizations developing, providing or using AI. Independent certification bodies assess and issue certification; ISO does not do so itself.

Certification says that an organization has an AI management framework and passed an audit. It does not guarantee the accuracy or fairness of a particular product, or replace legal compliance. Sales materials must state the certified scope accurately (organization, sites and activities). Using supplier certification in procurement still requires assessing risks of your own use cases.

Frontier safety frameworks

Some AI developers publish voluntary frameworks addressing severe risks of frontier models.

These frameworks aim to manage extreme capability risks potentially causing mass harm during model development and deployment. They inform procurement about supplier safety management. They are not data processing contracts, do not guarantee individual output accuracy or fairness, and do not transfer the adopting company's risk assessment or legal responsibilities. AI Basic Act Article 32 shares a concern with extremely high-compute frontier systems, but statutory obligations and voluntary frameworks have different grounds and effects.

Going deeper

The traps of scoring

A 5 × 5 impact and likelihood matrix has three traps. Combining different harms into one number obscures what needs prevention. Low-likelihood but irreversible risks disappear under low scores. Weak estimates appear factual once converted to numbers.

Record two things beside scores: uncertainty (the evidence underlying the assessment, and unmeasured groups or conditions) and conditions addressed regardless of score (legal obligations, irreversible harm, effects on children or vulnerable groups). Mark unmeasured aspects "Unmeasured", not "Low-risk", and inform recipients. Where key life, safety or rights metrics are unmeasured, the acceptance authority must explicitly decide whether to defer deployment.

Nondeterministic outputs and reproducibility

Generative AI selects subsequent words probabilistically, so identical inputs may yield different answers. Perfectly reproducing why a particular answer appeared is difficult. This does not mean, however, that explanation can be abandoned. Instead of exact reproduction, keep contextual evidence: request time, model and version, system prompt version, retrieved document identifiers, key settings, output and human review result. This collection can explain the conditions under which an answer arose even when the answer itself cannot be reproduced.

At work

Neulpum Foods: creating the first risk register

Neulpum Foods decides to register nine AI uses for the first time, including demand forecasting, production-line foreign-object image inspection, marketing copy, customer chatbots and supplier document summaries. The Corporate Administration team initially proposes marking everything high-risk because the company handles food safety.

The risk officer writes harm scenarios first. Missed foreign objects could injure consumers with difficult recovery, making inspection high-risk. AI supplements rather than replaces existing metal detectors and human sampling. Marketing copy is medium due to false or exaggerated claims, with review before publication. Supplier document summaries are low because they are internal drafts.

The customer chatbot divides opinions: incorrect allergy information could harm health. Rather than raise the whole chatbot to high-risk, the officer avoids that risk by routing allergy and ingredient questions to official ingredient lists and staff without generating chatbot responses. Delivery and order inquiries remain medium. Since it is a generative AI service, Article 31 notices and labels are separately marked as statutory requirements regardless of level.

The register records the acceptance authority's name, reassessment date and uncertainty: "Foreign-object inspection performance under nighttime lighting has not yet been measured."

Doran Doran: auditing mistaken automated sanctions

Community app Doran Doran uses AI to hide posts suspected of abuse or fraud automatically, temporarily suspending accounts on repetition. After a vendor model update last month, complaints increased that posts using regional dialect and older users' spacing habits were being hidden as suspected fraud.

The auditor first looked for change records. Fortunately, the company logged model version, classification criteria version and scores for each judgment. Comparing before and after revealed increased misclassification of certain expressions. Prior evaluation used mainly standard-language data and had not measured these group differences.

The auditor recommended three changes. First, account suspensions should require operator confirmation rather than automatic execution. The PIPC notice's schedule also treats temporary blocking after automatic detection followed by a final human decision on request as an exception to automated decisions. Second, notify users of hidden posts' reasons and appeal methods, with human review of appeals within seven days. Third, include model replacement in change management and validate new versions on regional and age-group samples before deployment. Developers' claim of better public benchmark scores did not alter the decision: those results were not measured on the company's users.

The report separated design problems (evaluation data representativeness) from operational problems (missing change management), attaching evidence of wrongly suspended account recovery and individual notices.

Common mistakes

1. "Maximum controls everywhere ensure safety." Resources are spread too thin to protect genuinely risky areas.
2. "Supplier certification lets us skip assessment." The adopting organization assesses risks in its own context.
3. "Frontier safety frameworks manage bias and errors too." These frameworks address extreme capability risks.
4. "AI RMF is completed once in order." Functions repeat, with Govern spanning the process.
5. "Contractual transfer eliminates risk." Costs can be shared, but harm to affected people remains.
6. "Low scores reduce legal obligations." Legal requirements apply regardless of score.
7. "Reproducible logs mean we have explained it." Reproduction is transparency; helping people understand reasons is interpretability.
8. "Overwrite documents with the latest version on model changes." Preserve version-specific records to explain past decisions.

Summary at a glance

Check the official docs

• NIST — AI Risk Management Framework: AI RMF 1.0, revision notice, Playbook and Generative AI Profile.
• NIST AIRC — AI RMF Core: Four functions and subcategories (MANAGE 1.2–1.4, 4.1, 4.3 and MEASURE 3.3).
• NIST AIRC — Trustworthy AI characteristics: Seven characteristics and distinctions among transparency, explainability and interpretability.
• NIST AI 600-1 — Generative AI Profile: Definitions of twelve risks.
• ISO — ISO/IEC 42001 explained: AI management systems and certification.
• Anthropic — Responsible Scaling Policy: Current version and revision history.
• Google DeepMind — Frontier Safety: Framework purpose and version history.
• OpenAI — Preparedness Framework (v2): Tracked categories and capability thresholds.

Self-check

5-1. Replace "hallucination risk" with one sentence describing a harm scenario, identifying affected people, conditions, harm and recoverability.

5-2. List NIST's seven trustworthy AI characteristics, identifying the necessary condition and the characteristic related to all others.

5-3. Name five of NIST AI 600-1's twelve risk categories, and explain in one sentence how to use the list when building a risk register.

5-4. List the four risk response choices, and explain why major human-impact risks cannot be handled solely by contractual transfer.

5-5. Name two records covered in this chapter whose retention is legally specified under the AI Basic Act or PIPA, and give the periods.

5-6. Contrast two things an AI management system standard certificate demonstrates with two things it cannot demonstrate.

5-7. (Design) Invent four AI use cases and build a risk register. Each row should include a harm scenario, impact, likelihood, level, controls, basis (law, contract or internal), acceptance authority and uncertainty.

Hands-on tasks