Chapter 4. Model and Data Risk Verification

Learning objectives

• Calculate accuracy, precision, recall, and false positive rate, and compare them with appropriate baselines.
• Verify model performance claims based on evaluation set representativeness, separation from training, and measurement timing.
• Disaggregate performance by subgroup and verify the presence of bias by considering the uncertainty of small-sample groups and skewed ground-truth labels.
• Examine the provenance, label quality, and augmentation status of training and operational data.
• Determine whether reverification is required and its scope when model modifications or data shifts occur.

Introduction

In an executive reporting document, a bank's data team wrote: "The accuracy of the new fraudulent transaction detection model is 99.8%." While the number was impressive, an auditor immediately asked several questions: What percentage of transactions are fraudulent? If fraud accounts for 0.1%, a "model" that classifies every transaction as legitimate would achieve an accuracy of 99.9%. In that case, 99.8% might be a figure worse than doing nothing at all.

The most quintessentially AI risks in an AI system stem from the model and data. A model guarantees its performance only under the conditions present during evaluation, data changes over time, and a model trained on past human decisions learns even the biases embedded in those decisions. Auditors do not build models themselves, but they must be able to judge whether claims about a model are supported by evidence. To do so, rather than complex mathematics, one needs the habit of persistently questioning what the numbers were compared against and under what conditions they were measured.

This chapter begins by introducing the vocabulary of performance—namely, the meanings of validity, reliability, and accuracy, along with confusion matrix metrics and baselines. It then covers the criteria an evaluation set must satisfy and quality measurement for generative AI, followed by an examination of bias, subgroup performance, and data provenance and quality. Finally, it outlines operational changes and the scope of reverification required after model modifications.

Key concepts

Validity, reliability, accuracy, and robustness

The NIST AI RMF places "valid and reliable" among the characteristics of trustworthy AI as a necessary prerequisite underpinning the other characteristics. The terms are defined as follows:

• Validation: Confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled (cited by NIST from ISO 9000:2015).
• Reliability: The ability of an item to perform as required, without failure, for a given time interval, under given conditions.
• Accuracy: Closeness of results of observations, computations, or estimates to the true values or the values accepted as being true.
• Robustness: The ability of a system to maintain its level of performance under a variety of circumstances.

NIST explains that measuring accuracy requires considering computational metrics such as false positive and false negative rates, performance when humans and AI work together, and generalizability beyond training conditions, and notes that accuracy measurements should always be presented alongside clear and realistic evaluation sets that represent real-world operating conditions, together with details of the testing methodology. It also advises that results may be broken down across data segments and that consideration should be given to the fact that certain failures cause greater harm.

Confusion matrix and metrics

The outcomes of a classification model are organized into four cells. Taking fraudulent transaction detection as an example:

The metrics are derived from these four cells.

Let us practice with numbers. Suppose that out of 500,000 transactions, 500 were actual fraud, and the model flagged 600 transactions as fraud, of which 450 were actual fraud. In this case, TP is 450, FP is 150, FN is 50, and TN is 499,350. Recall is 450 ÷ 500 = 90% (the false negative rate is 50 ÷ 500 = 10%), precision is 450 ÷ 600 = 75%, the false positive rate is 150 ÷ 499,500 ≈ 0.03% (specificity is approximately 99.97%), and accuracy is 499,800 ÷ 500,000 = 99.96%. However, even if every transaction were classified as legitimate, the accuracy would still be 499,500 ÷ 500,000 = 99.9%. In this example, accuracy reveals almost nothing about the usefulness of the model, whereas recall and precision convey the essential insights.

Which metric is important is determined by the cost of errors. Missing a fraud case leads to financial loss (false negative), whereas blocking a legitimate transaction causes customer inconvenience (false positive). Where the cost of missing cases is severe, such as in medical screening, recall is critical; where the cost of blocking legitimate customers is high, precision and the false positive rate become paramount. Remember that these two metrics have different denominators. When legitimate transactions are overwhelmingly dominant, the false positive rate appears very small, such as 0.03%, but in the same scenario, a precision of 75% means that only three out of four alerts are actual fraud, while one represents an erroneous disruption to a legitimate customer. Customer inconvenience and investigation workload are much better reflected by precision.

Baselines

Performance figures become meaningful only when there is an object of comparison. Auditors rely on three types of baselines:

1. Majority-class baseline: Performance achieved when every case is classified as the most frequent outcome. This must be examined when interpreting accuracy in models designed to detect rare events.
2. Incumbent-method baseline: The performance of the rule-based system or human operators prior to AI adoption. This is used when verifying the claim that "AI has improved performance."
3. Human baseline: The performance when human operators evaluate the exact same cases. This is used when verifying the claim that the model is "more accurate than humans."

When comparing against a baseline, one must use the same cases, the same time period, and the same scoring criteria. A table that places a human error rate measured three years ago on different cases alongside an AI error rate measured on recent data does not constitute a valid comparison.

Criteria required of an evaluation set

The NIST AI RMF calls for documenting the evaluation sets, metrics, and tools used in test, evaluation, verification, and validation (TEVV) (MEASURE 2.1), demonstrating performance under conditions similar to the deployment environment (MEASURE 2.3), and demonstrating that the system to be deployed is valid and reliable while documenting the limitations of generalization beyond the development conditions (MEASURE 2.5). Auditors ask five questions regarding an evaluation set:

1. Separation: Were the evaluation cases excluded from training? If records from the same customer or account are split across training and evaluation sets, the model may memorize individual characteristics and achieve higher scores than it would on previously unseen customers. This issue is called data leakage.
2. Representativeness: Does it reflect actual operating conditions, such as the distribution of transaction amounts, inquiry types, customer demographics, and seasonality? A set biased toward easy cases overestimates operational performance.
3. Timing: When was the measurement taken? If the product mix or customer base changed significantly after the measurement, historical figures do not serve as evidence of current performance.
4. Label quality: Who assigned the ground-truth labels, and how? If labels are incorrect or inconsistent, the integrity of all metrics is compromised.
5. Metric suitability: Does the metric align with the costs of errors? If only accuracy was reported for rare events, the metrics must be recalculated.

Measuring the quality of generative AI

Because the outputs of generative AI often lack a single definitive correct answer, measuring their quality is more difficult. NIST AI 600-1 identifies confabulation—the generation of confident-sounding but incorrect or fabricated content (commonly termed hallucination)—as a distinct risk of generative AI. Auditors verify the following:

• Cross-checking against authoritative sources: Satisfaction ratings or "helpful" rates may reflect plausibility rather than the factual accuracy of responses. Accuracy must be measured by cross-checking against official authoritative materials.
• Segmented measurement by question category: High-stakes inquiry categories where errors cause severe harm—such as interest rates, eligibility requirements, or medical information—must be measured separately so that risks are not obscured within aggregate averages.
• Interpretation of human modification rates: The rate at which customer service agents sent responses without edits does not equate to draft accuracy. Due to automation bias, incorrect drafts may be dispatched without modification.
• Documentation of evaluation methodology: The specific question set, scoring rubrics, and identities or qualifications of the evaluators must be documented to enable replication and comparison of results.

Three categories of bias

NIST categorizes AI bias into three categories. Systemic bias can be present in datasets, organizational norms and practices, and broader society where AI systems operate. Computational and statistical bias arises from the processing of datasets and algorithms, frequently stemming from systematic errors in unrepresentative samples. Human cognitive bias stems from the ways individuals perceive and interpret AI outputs or make judgments during system design and operation. Furthermore, NIST explains that all three categories can occur in the absence of explicit prejudice or discriminatory intent.

This distinction carries significant implications for auditing. A development team's explanation that "there was no intent to discriminate" does not constitute grounds to conclude an audit review of bias. Furthermore, NIST points out that a system in which harmful bias has been mitigated is not necessarily fair, as predictions that achieve a degree of balance across groups may still remain inaccessible to persons with disabilities or exacerbate existing disparities.

Subgroup performance verification

The NIST AI RMF instructs organizations to evaluate fairness and bias and document the results (MEASURE 2.11), while the GAO framework calls for assessing the reliability, quality, representativeness, and potential bias of data used in operations (Practice 2.7) and identifying potential bias and inequities resulting from system outputs (Practice 3.8). The foundation of subgroup verification is disaggregating results: even when overall accuracy is high, the error rate for a specific subgroup may be elevated.

Three challenges arise when analyzing disaggregated results:

1. Small-sample subgroups. In a subgroup consisting of only a few dozen cases, even large disparities may arise by chance. The same principle applies here as in audit sampling, where the risk that conclusions drawn from a sample may differ from the conclusion reached if the entire population were tested (sampling risk) increases as the sample size decreases (AS 2315 .10). Auditors should extend the observation window to collect more cases and report the observed disparity alongside its uncertainty (confidence interval). Small sample sizes must not simply be disregarded.
2. Proxy variables. Even if gender or age is excluded from the input features, other variables such as place of residence, occupation, or usage patterns may act as proxies. For example, if a specific age group is heavily concentrated in a particular neighborhood, residential location serves as a proxy for age. The assertion that "bias is absent because sensitive variables were excluded" is invalid.
3. Bias in ground-truth labels. A model trained on the approval and rejection decisions of past reviewers inherits the historical bias embedded in those determinations. In this context, a high agreement rate with historical decisions does not demonstrate fairness; rather, it may simply indicate that the model faithfully replicates past biases. Labels must be vetted against independent outcomes, such as actual repayment performance.

Data provenance and lineage

The GAO framework instructs organizations to document the sources and provenance of data used in model development (Practice 2.1), assess data reliability (2.2), categorize data attributes (2.3), evaluate variables used in the model (2.4), and assess the use of synthetic, alternative, or augmented data (2.5). The NIST AI RMF calls for mapping legal risks and the risk of infringing third-party intellectual property associated with system components, including third-party data and software (MAP 4.1).

A vendor's claim of "guaranteed 99% accuracy" or the assertion that "anyone can use it because it is public data" does not constitute evidence of quality or legality. Public data is governed by licenses and terms of use, and vendor warranties represent claims made by an interested party.

Label quality

The performance of a supervised learning model cannot exceed the quality of its ground-truth labels. The NIST AI RMF calls for identifying and documenting scientific integrity and TEVV considerations, including data collection and selection (availability, representativeness, suitability) and concept validity (MAP 2.3). Practical methods for inspecting label quality include:

1. Inter-annotator agreement rate: Having multiple individuals annotate the same sample independently to measure the degree of agreement. A low agreement rate indicates ambiguous criteria or inadequate training.
2. Consistency of guideline documentation: When multiple branches or teams assigned labels, compare whether they used identical guideline documents.
3. Ground-truth re-review sample: Having subject matter experts re-evaluate a sample of labels to measure the error rate.

Adding legacy data whose standards cannot be verified simply to increase volume may degrade data quality.

Operational shifts and monitoring

A model begins to degrade from the very day it is deployed. This is because data drift occurs when the distribution of input data changes, and concept drift occurs when the relationship between inputs and outputs shifts. The GAO framework advises establishing a continuous monitoring plan (Practice 4.1), defining acceptable thresholds for data and model drift (4.2), and documenting monitoring results and corrective actions (4.3). The NIST AI RMF likewise calls for monitoring system functionality and behavior in operation (MEASURE 2.4) and incorporating user feedback collection, appeals and reversals, incident response, recovery, and change management into post-deployment monitoring plans (MANAGE 4.1).

In monitoring controls, what auditors examine is not whether "alerts exist," but rather "who does what when an alert triggers." A control fulfills its purpose only when alerting criteria, recipients, response procedures, and records of action are all present. If alerts triggered fourteen times without a single recorded corrective action, the alert mechanism functioned, but the control itself failed to operate.

Change types and scope of reverification

The scope of reverification is proportional to what has changed and the extent to which it affects outputs.

Factors modified by vendors must be viewed through the same lens. The NIST AI RMF calls for incorporating pre-trained models used in development into regular monitoring and maintenance (MANAGE 3.2). A vendor's publicly published benchmark scores do not reflect performance on your organization's specific questions and data.

Reverification procedures and approval criteria

An effective reverification process follows a defined sequence: document the modifications and the scope of impact, compare the versions before and after the change using the same evaluation set, evaluate the results against predefined approval criteria to either approve or reject deployment, and update post-deployment monitoring baselines to match the new version. The NIST AI RMF calls for determining whether an AI system achieves its intended purpose and objectives and deciding whether to proceed with development or deployment (MANAGE 1.1).

When approval criteria are established only after viewing the results, reverification easily deteriorates into a formality. Auditors verify whether approval criteria were documented before the results were known, whether the criteria include subgroup metrics, and whether proposed changes that failed to meet the criteria were actually rejected in practice.

Going deeper

There is no single standard for fairness

To the question "Is this model fair?", there is no single correct metric. Equalizing approval rates across groups, equalizing error rates across groups, and assigning the same meaning to the same score are different concepts of fairness, and satisfying one can cause another to fall out of alignment. NIST also explains that standards of fairness are complex and difficult to define, and perceptions of fairness vary across cultures and application domains.

Therefore, it is dangerous for an auditor to establish standards of fairness independently. Standards must be determined and documented by management after considering laws and regulations, the nature of the business, and the perspectives of affected individuals. Auditors examine whether those standards exist, what was considered when establishing them, whether measurements are performed according to the standards, and what actions were taken when standards were breached. If there are no standards, that in itself becomes a finding. If there is a numeric standard such as "keep the difference within 3 percentage points," the auditor verifies the data and calculations used to apply that figure.

Performance when humans and AI work together

Many AI systems assist human judgment rather than replace it. In this context, what matters is not the performance of the AI in isolation, but the performance when humans and AI work together. This is why NIST recommends taking human-AI collaboration into account when measuring accuracy.

Performance when working together can be better or worse than AI alone. It improves if humans filter out AI errors effectively; it remains the same as AI alone if automation bias leads humans to accept AI errors uncritically; or it becomes worse if humans overturn correct AI judgments without justification (algorithm aversion, as described in NIST AI 600-1). Faced with a decision such as "AI is more accurate than humans, so let us reduce the frequency of inspections," an auditor can request three figures: the results of humans alone, AI alone, and humans and AI working together on the same cases. Only with these three figures can one determine what is sacrificed by the reduction in inspections being pursued.

At work

Breeze Bank: Rereading the '99.8% Accuracy' report

The digital banking department of Breeze Bank issued a report stating that a new wire transfer fraud detection model was "superior to the existing rule-based system (99.5%) with a 99.8% Accuracy." The audit team requested the raw data before using the report as evidence.

The evaluation data comprised 1.2 million wire transfers over the preceding six months, of which 960 were fraud (0.08%). When the audit team reconstructed the confusion matrix, the new model caught 610 out of the 960 fraud cases (recall of approximately 64%), and 610 out of the 2,700 flagged cases were actual fraud (precision of approximately 23%). The existing rule system caught 480 fraud cases over the same period (50% recall). If all transfers were classified as normal, accuracy would be 99.92%, meaning that on accuracy alone, both systems performed worse than the majority-class baseline—that is, "doing nothing." The audit team recommended removing the accuracy comparison from the report and instead comparing recall, precision, and the number of normal customer transfers blocked across the same period and identical cases. While the new model was clearly superior in recall, it also blocked more legitimate transfers.

The audit team also reviewed the evaluation design. The development team had split the transfer records into training and evaluation sets by shuffling them randomly, resulting in transfers from the same fraudulent accounts appearing in both sets. The audit team required them to re-evaluate by splitting on an account basis, and recall dropped from 64% to 57%. Finally, when breaking down results by age group, the rate of normal transfers blocked for customers aged 70 or older was twice that of other age groups. This group had a sufficient volume of just over 10,000 transfers during the six-month period. The bank did not yet have a standard regarding disparities across groups. The audit team cited the absence of a standard as a finding, noting the operational impact of false positives on elderly customers (transfer delays, call center verification procedures).

United Express: One document, one model, one threshold

The parcel delivery company United Express used AI to classify delivery complaint inquiries, generative AI to draft responses, and a separate predictive model to identify and pre-inspect shipments with a high risk of loss. The audit team reviewed the re-verification records for three changes that occurred in the previous quarter.

First, compensation regulations were revised, and among the knowledge documents for the draft-response AI, only the Loss Compensation Guide was updated. The operations team submitted only a change approval form. Noting that compensation-related inquiries accounted for 15% of the total, the audit team requested regression evaluation records comparing responses before and after the change on compensation queries. Because no records existed, the audit team tested a sample of compensation inquiries directly and found mixed responses that failed to reflect the new regulations. This occurred because, separate from the updated document, an FAQ document containing the previous regulations remained in the retrieval scope.

Second, the foundation model for the classification AI was replaced with a newer version from the same vendor. The operations team submitted documentation stating that "vendor benchmark scores improved." Noting that replacing a foundation model carries an organization-wide scope of impact, the audit team required a full re-evaluation using the company's inquiry data. The operations team had conducted an evaluation comparing classification accuracy by inquiry type prior to the changeover, but lacked acceptance criteria and approved it after reviewing the evaluation results on the basis of "no significant difference." The audit team recorded the absence of predetermined acceptance criteria.

Third, the decision threshold score for the loss risk model was lowered from 0.7 to 0.6. As the volume of inspected shipments grew, losses decreased, but the workload on field inspection staff rose by 40%. The data team maintained before-and-after comparisons of recall and precision across the entire dataset and by regional hub, and had established acceptance criteria in advance (an overall recall of 85% or higher, no hub dropping below 70%, and a daily inspection volume ceiling). The audit team highlighted this change in its report as a good example of re-verification procedures and recommended that the other two changes also adopt predetermined criteria and segmented comparisons.

Common mistakes

1. "If accuracy is high, it is a good model." Models that detect rare events must be compared against the majority-class baseline. Recall and precision tell the essential story.
2. "It was evaluated on data not used for training, so there is no problem." If records from the same customer or account are split across both sets, they are not effectively separated.
3. "The performance measured at launch is still the current performance." If the product mix or customer base has changed, historical figures are not evidence of current performance.
4. "The rate at which agents sent drafts without edits is the draft's accuracy." The unedited transmission rate merely indicates the degree to which humans complied with the AI.
5. "Since sensitive variables were excluded from the input, there is no bias." Other variables can act as proxies, and the labels themselves may be skewed.
6. "Disparities in groups with few cases can be ignored." Determine first whether the variation is due to chance, and verify by extending the observation period. Do not discard a group merely because it is small.
7. "Since the vendor guaranteed quality, the data can be trusted." A guarantee is an assertion made by an interested party. Reliability must be verified independently.
8. "Since vendor benchmark scores increased, re-verification is not needed." The questions and data distribution of a benchmark differ from our operational tasks, so they do not demonstrate performance on our data and queries.

Summary at a glance

Check the official docs

• NIST AI RMF — AI Risks and Trustworthiness: Definitions of validity, reliability, accuracy, and robustness; three categories of bias.
• NIST AI RMF — Core: MAP 2.3, 4.1; MEASURE 2.1, 2.3, 2.4, 2.5, 2.11; MANAGE 1.1, 3.2, 4.1.
• NIST AI 600-1: Confabulation, Harmful Bias or Homogenization, data provenance information.
• GAO-21-519SP: Data (2.1–2.8), Performance (3.1–3.9), Monitoring (4.1–4.5) principles and audit questions.
• PCAOB — AS 2315: Relationship between sampling risk and sample size (.10).

Self-check

4-1. Out of 200,000 transactions, 200 were actual fraud, and the model flagged 300 transactions as fraud, of which 150 were actual fraud. Calculate the recall, precision, and accuracy of the majority-class baseline.

4-2. Write one sentence each explaining the difference between validation and reliability as defined by NIST.

4-3. Among the five questions an auditor asks regarding an evaluation set, select the one that becomes particularly problematic when training and evaluation sets are split by randomly shuffling rows, and state the reason.

4-4. State the three categories of bias described by NIST, and explain why the explanation "the development team had no intention to discriminate" cannot serve as grounds for concluding a bias review.

4-5. In a model trained on past decisions by human staff as ground truth, explain why "a 95% concordance rate with past decisions" does not serve as evidence of fairness.

4-6. State the appropriate re-verification scope for each of the following: adjusting decision threshold scores, updating part of the knowledge documents, and replacing the foundation model.

4-7. (Design) Design a quality measurement plan for a service where generative AI provides guidance on eligibility for public subsidies. Include the composition of the evaluation question set, scoring criteria, question types to measure separately, and operational monitoring metrics.

Hands-on tasks