Chapter 5. Fairness Auditing and Governance

Learning objectives

• Calculate selection rates and selection rate ratios, and distinguish between what the four-fifths rule says and what it does not say.
• Interpret outcome differences between groups by considering sample size and statistical significance, and locate where disparities arise through stage-by-stage analysis.
• Select appropriate actions based on audit results, and identify flawed actions such as group-specific cutoff scores.
• Define the cycles and triggers for periodic audits and change-triggered audits, and select validation materials to request from vendors.
• Design the framework for the accountability of hiring AI and operating rules using the four functions of the NIST AI RMF.

Introduction

Suppose a line in a company's quarterly report reads: "AI resume screening accuracy 92%, target achieved." In that same quarter, the pass rate for female applicants was half that of male applicants, and no one calculated that number. Another company, seeing that only one out of nine applicants with disabilities passed, immediately proposed discarding the tool, claiming that "the tool is biased." The former company failed to look, while the latter drew far too grand a conclusion from far too small a number.

Fairness audits are harder to interpret than to calculate: which groups to compare, which stages to examine, whether the sample size is sufficient, whether the disparity is accidental, what caused the disparity, and what should be done. And if the personnel and procedures to answer these questions are not established, audit findings remain confined to reports.

This chapter begins with numbers and ends with the organization. First, you will learn selection rates and the four-fifths rule, sample size and significance, and stage-by-stage analysis. Next, it covers how to turn audit results into actions, audit schedules, and vendor validation materials. Finally, aligned with the four functions of the NIST AI RMF, it establishes who is accountable for what and formulates operating rules.

Key concepts

What fairness measures

The NIST AI RMF lists seven characteristics of trustworthy AI: Valid and Reliable, Safe, Secure and Resilient, Accountable and Transparent, Explainable and Interpretable, Privacy-Enhanced (protecting personal information), and Fair – with Harmful Bias Managed. NIST notes that there are usually trade-offs among these characteristics, meaning that all characteristics rarely apply equally across all contexts.

NIST emphasizes two key points regarding fairness. First, bias originates not only from data and algorithms, but also across systemic, computational and statistical, and human-cognitive categories. Second, a system that has reduced harmful bias is not necessarily fair. A single metric exceeding a threshold does not confirm fairness; auditing is the work of generating material for judgment.

In hiring AI, the most fundamental fairness audit consists of examining differences in outcomes: how much the advancement rates to the next stage differ by group, and whether that difference can be explained by job-related reasons. This chapter proceeds centered on these questions.

Selection rate and selection rate ratio

The selection rate is the proportion of applicants within a group who advance to the next stage. The selection rate ratio is the selection rate of a comparison group divided by the selection rate of the group with the highest selection rate.

In this example, the selection rate ratio for Group B is 22 ÷ 30 ≈ 0.73. While the calculation is simple, one must first decide what constitutes a group (gender, age group, disability status) and which stage of passing to examine (document screening, aptitude test, interview, final).

The four-fifths rule: What it says and what it does not say

The Uniform Guidelines on Employee Selection Procedures of U.S. federal agencies (29 CFR 1607.4(D)) establishes a rule of thumb stating that a selection rate for any race, sex, or ethnic group which is less than four-fifths (80%) of the rate for the group with the highest rate will generally be regarded by federal enforcement agencies as evidence of adverse impact, while greater than four-fifths will generally not be regarded as evidence. This book introduces this not as a law applicable to Korean companies, but as a starting point for interpreting outcome differences.

The same provision also clearly notes the limitations of this rule of thumb.

In Korean law, the same inquiry leads to statutory provisions on indirect discrimination. Article 2, Subparagraph 1 of the Equal Employment Opportunity and Work-Family Balance Assistance Act defines it as discrimination when, even if the same conditions are applied, the proportion of one sex capable of satisfying those conditions is substantially smaller than that of the other sex, resulting in an adverse outcome for a specific sex, and the employer cannot prove that the conditions are justified. Article 4-4, Paragraph 2 of the Act on Prohibition of Age Discrimination in Employment and Elderly Employment Promotion likewise considers it age discrimination if applying criteria other than age without reasonable grounds produces a particularly disadvantageous outcome for a specific age group. Therefore, the four-fifths rule is used as a signal to find "whether a problem exists," not as a tool to prove that "it is not a problem."

Sample size and statistical significance

Statistical significance refers to the degree to which an observed difference is unlikely to have arisen by chance alone. Let us understand this intuitively. If a coin is flipped ten times and heads appears seven times, it is difficult to conclude that the coin is biased. However, if it is tossed ten thousand times and heads appears seven thousand times, it can hardly be seen as a coincidence.

In practice, the difference in pass rates between two groups is verified using a chi-square test, and when the number of individuals is small enough that any cell in the contingency table has an expected frequency of less than 5, Fisher's exact test is used. If it is difficult to determine the method, document it in advance in the audit plan together with a statistician. Safeguards are needed to mitigate harm to applicants while conclusions are deferred due to small sample sizes. An example is having humans re-review rejection cases from the group in question. "Doing nothing because the sample is small" is different from "deferring judgment because the sample is small while preventing harm."

Stage-by-stage analysis

Looking only at final pass rates makes it impossible to know where differences arose. Because hiring goes through multiple stages, pass rates for each group must be calculated separately at each stage to distinguish whether the AI document screening is the cause or the human-conducted interview is the cause.

The pass rate for each stage is the ratio of individuals who passed relative to the number of individuals who were evaluated at that stage.

In this example, one must examine the interview stage before the AI document screening. Fairness auditing is not merely about looking at the AI, but about looking at the entire hiring process.

Group disparities in errors

In addition to pass rates, differences in errors are also examined. Check whether errors where the AI underestimates applicants who actually meet job requirements (misses) are concentrated in a specific group. Within a sample re-reviewed by humans, calculate the proportion of applicants rejected by the AI among those "judged by humans to meet requirements" for each group. Because group sizes differ, comparisons must be made using proportions rather than counts. Because multiple fairness criteria (balance of pass rates, balance of error rates, etc.) are often difficult to satisfy simultaneously, organizations must document which criteria take precedence and the reasons why. This is where the trade-offs among characteristics described by NIST appear in concrete form. In particular, it is widely known that when the true qualification rates (base rates) of two groups differ, it is generally impossible to satisfy both a balance of pass rates and a balance of error rates simultaneously, so this choice must be determined in advance.

Turning audit findings into action

Because the cause differs for each audit finding, the corrective actions must also differ.

Setting different passing cutoff scores for each group to balance audit metrics is not an appropriate action. This is because it treats individuals differently based on their group while leaving the underlying cause untouched. The principle is to identify and correct the items or stages that created the difference. Preferential systems established with defined legal grounds and objectives—such as affirmative action programs or statutory support measures—are a distinct matter; if operating such programs, verify their legal basis separately.

Audit cycles and triggers

Fairness audit findings are valid only for the specific system audited. Therefore, both cycles and triggers must be defined together.

1. Periodic audits: Conducted quarterly or semi-annually using the same methodology. Document findings to compare trends.
2. Change-triggered audits: Conducted using the same methodology, separate from periodic audits, whenever model versions, input features, cutoff scores, or weights change.
3. Anomaly-triggered audits: Conducted immediately upon sudden shifts in pass rates, an increase in appeals, or reviewer reports.

In particular, pre-change audits must verify whether newly added input features—such as external activity records—are items in which specific groups might find it difficult to participate. Because NIST AI RMF MANAGE 3.1 requires organizations to regularly monitor and apply controls to the risks and effectiveness of third-party resources, a vendor's model update is itself a trigger for an audit.

Group data for auditing

Calculating group-level results requires group demographic data. However, collecting information for auditing also constitutes processing of personal information. Health-related data, such as disability status, can qualify as sensitive data (Article 23 of the Personal Information Protection Act (PIPA)). Do not make such data mandatory solely for fairness auditing purposes; absent statutory grounds, obtain separate consent distinct from other processing, ensuring there are no adverse consequences for refusal. The principles are as follows: Do not collect new data if calculations can be made with already acquired information. If new data must be collected, disclose the audit purpose, collect it voluntarily, and ensure candidates suffer no disadvantage if they decline. Store auditing data separately from the evaluation system and ensure evaluators cannot view it.

Requesting vendor validation data

Even when using vendor AI, the accountability of risk management also rests with the using company. NIST AI RMF GOVERN 6.1 requires policies and procedures to address AI risks associated with third parties. Materials to request during contracting and renewal are as follows:

A single line stating "overall accuracy," customer satisfaction ratings, or award histories do not answer these questions. A letter of commitment stating that "the vendor assumes all liability" also does not eliminate the auditing accountability of the company. Because operators providing products or services utilizing high-impact artificial intelligence must endeavor to conduct fundamental rights impact assessments (Article 35 of the AI Basic Act), and state agencies and other public institutions must give preferential consideration to products that have undergone such assessments, inquire with vendors about whether an impact assessment was conducted and request a summary of the results.

The four functions of the NIST AI RMF

The Core of the NIST AI RMF, a voluntary framework, divides AI risk management into four functions. GOVERN is the function that cultivates a culture and accountability of risk management, spanning across all three other functions. MAP identifies the context of the AI system and categorizes risks. MEASURE analyzes and assesses risks using quantitative and qualitative methods. MANAGE allocates resources to respond to identified and measured risks. While NIST explains that the functions can be carried out in any order, most users build upon GOVERN, begin with MAP, and proceed to MEASURE and MANAGE.

Accountability Framework and Operating Rules

NIST AI RMF GOVERN 2.1 calls for documenting roles, responsibilities, and communication channels regarding the identification, measurement, and management of AI risks, while GOVERN 2.3 ensures that organizational executive leadership takes responsibility for decisions concerning AI risks. Operating rules for hiring AI must include at least the following:

1. Operations lead: One person per tool. Distinct from the vendor liaison.
2. Change control: Who can modify and who approves baselines, weights, input features, and model versions. Maintain change records (roles under GOVERN 2.1, change management under MANAGE 4.1).
3. Suspension authority and criteria: Upon which signals, who halts tool use and transitions to human evaluation (MANAGE 2.4).
4. Audit schedule: The cycles and the person in charge, and reporting lines for periodic audits, change-triggered audits, and anomaly-triggered audits (GOVERN 1.5).
5. Appeals channel: The channel and resolution timeframe for applicants to contest outcomes, along with procedures to integrate appeal contents into audits (MEASURE 3.3, GOVERN 5.1). If it is an Automated Decision, align this channel with the refusal, explanation, and review request procedures and the 30-day deadline from Chapter 2.
6. Records and retention: Retention periods and access permissions for audit findings, decision logs, and change records.

Because establishing or substantially modifying a hiring AI system may involve matters subject to consultation by the labor-management council (Article 20, Paragraph 1, Subparagraphs 2, 5, and 9 of the Act on the Promotion of Worker Participation and Cooperation), specify the timing of consultation in the operating rules as well.

Incident response

When a signal emerges indicating active harm to real applicants, such as a sharp shift in pass rates, the sequence is clear: first halt use (switch to human review or a previous version), re-evaluate affected applicants, investigate the cause, and establish preventive measures. NIST AI RMF MANAGE 4.3 calls for communicating incidents and errors to relevant actors, including affected individuals. Conversely, if the same issue was identified during testing prior to live deployment with actual applicants, because there are neither live operations to halt nor applicants to re-evaluate, the core tasks are root-cause analysis, remediation, and re-validation under the same criteria. Furthermore, if a configuration change was discovered and rolled back before being reflected in outcomes, the core task is not re-evaluation, but closing gaps in change control.

Going deeper

Why group-specific cutoff scores are not the solution

The fastest way to eliminate differences in outcomes is to set different cutoff scores for each group. However, this is not a solution for two reasons. First, the causes that created the difference (proxy variables, unverified metrics, biased interviews) remain untouched. If you only adjust the numbers to match, the same problem repeats in the next recruitment cycle, and the motivation to identify the root cause disappears. Second, applying different criteria on the basis of gender or age is itself treating individuals differently because of those characteristics. Statutory exceptions, such as affirmative action under the Equal Employment Opportunity and Work-Family Balance Assistance Act, have defined purposes and characteristics, and therefore cannot be viewed as equivalent to adjusting criteria to balance audit numbers. The principle is always to find and fix the cause.

Good fairness metrics are not enough

Even if pass rates across groups are similar, if a tool is measuring something irrelevant to the job, it is an equally bad tool for all groups. Fairness stands on validity. Evidence that scores relate to job performance (MEASURE 2.5) comes first, and group-by-group outcomes (MEASURE 2.11) are examined on top of that. Validity evidence is built in two tracks: evidence that requirements correspond to tasks derived from job analysis (content validity), and evidence that scores relate to actual performance (criterion validity). Conversely, if a validated requirement produces unfavorable outcomes for a specific group, one must document the business necessity of that requirement, examine whether there is a way to achieve the same objective in a less exclusionary manner, and continuously monitor the outcomes. If these two questions are not separated, one falls into the same kind of misconception: "because it is fair, it is a good tool" or "because it is valid, it is a fair tool."

Governance is not a one-time approval

Many organizations view an AI adoption approval meeting as governance. The NIST AI RMF views GOVERN as a function that is continuously required across the entire life cycle of an AI system and that must be integrated into other functions. This is especially true for recruitment AI. Applicant demographics change every year, vendors update models, and job requirements change as well. Verification results from the time of adoption quietly become outdated over time. Therefore, the core of governance is not an approval document, but an inventory (what is being used), responsible persons (who is overseeing it), triggers (when it is reviewed again), and the authority to halt (who can stop it).

At work

Summer Industries' audit meeting

At a quarterly audit, industrial materials company Summer Industries received results showing that only one out of nine applicants with disabilities passed the AI resume screening. The pass rate for applicants without disabilities was 35%. Opinions were divided at the meeting. One side argued, "The selection rate ratio is about 0.32, so let us discard the tool immediately," while the other side argued, "Nine people is too small to be meaningful, so let us move on."

The audit lead did not accept either opinion. They acknowledged that drawing a conclusion was difficult because the percentage fluctuated significantly with only nine people. However, instead of moving on, they proposed three things: to aggregate data from the past four quarters within this week, recalculate, and check statistical significance; to have humans re-evaluate rejected resume screenings of applicants with disabilities in the ongoing hiring process until the analysis was completed and actions were determined; and to examine item by item how characteristics commonly found in applications from candidates with disabilities (e.g., career gaps, specific formats of personal statements) affected scores. They also verified whether accommodation request channels were properly communicated for online assessments and interviews.

When four quarters were pooled, 5 out of 41 applicants with disabilities (12%) passed, while 1,260 out of 3,600 applicants without disabilities (35%) passed, yielding a selection rate ratio of approximately 0.35; because there were only 5 passing applicants with disabilities, resulting in cells with low expected frequencies, Fisher's exact test was used, and the result also indicated a difference that could hardly be considered coincidental. In the item analysis, the variable "continuous employment period over the last two years" played a major role, and the job relevance of this variable had never been verified. The company decided to exclude this variable from rejection decisions until it was validated, and to remeasure the results in the same manner after the change. The discarded option was "let us lower the cutoff score only for applicants with disabilities." This was because that approach would have left the causal variable untouched.

Establishing operating rules at Well Bucket Construction

Construction company Well Bucket Construction used three AI tools for recruitment (resume screening, aptitude test scoring, and interview summarization), but each adopting department managed them separately. One day, an incident occurred in which no one could explain why the automatic rejection threshold for the resume screening tool had been changed. One staff member had changed it alone under schedule pressure, and fortunately, it was discovered before the changed criteria were applied to any applicant evaluations.

The HR team used this event as an opportunity to establish operating rules. First, they created an inventory of recruitment AI tools, organizing the purpose, vendor, version, person responsible, and next audit date for the three tools in a single table (GOVERN 1.6). Second, they designated an operational lead for each tool and required that any changes to thresholds, weights, input fields, or model versions undergo a request by the operational lead and approval from the HR executive, with a record of changes maintained (GOVERN 2.1, MANAGE 4.1). Third, they enabled the operational lead to immediately switch to human review if the pass rate changed by more than a specified margin compared to the previous quarter (MANAGE 2.4). Fourth, they included terms for prior notification of model changes and maintenance of previous versions in vendor renewal contracts (GOVERN 6.1, MANAGE 3.1). Fifth, they published the applicant appeals channel and processing deadlines in the recruitment notice, and required that the contents of appeals be placed on the quarterly audit agenda (MEASURE 3.3).

During this process, there was an opinion that "since no harm was done this time, let us conclude this with training for the person in charge." However, because the root cause was not an individual but a structure that had not defined who was authorized to make changes, they recognized that training alone would allow another employee to repeat the same mistake, and chose to establish rules instead.

Common mistakes

• "Overall accuracy exceeded the target, so it is fair." Overall accuracy does not show outcomes by group.
• "It exceeded 4/5, so there is no problem." In large samples, even small differences that are statistically and practically meaningful can constitute adverse impact.
• "It falls short of 4/5, so we should discard the tool." When the sample is small, withhold conclusions, aggregate data to examine it again, and put mechanisms in place to prevent harm in the meantime.
• "We only need to look at the final pass rate." You must divide it by stage to distinguish whether the cause is AI or human evaluators.
• "Setting different cutoff scores for each group resolves the issue." Adjusting criteria to balance audit numbers leaves the root cause intact and treats individuals differently on the basis of their group. Distinguish this from statutory affirmative action programs.
• "Since it is for auditing, collecting disability status is mandatory." Sensitive information must not be made mandatory solely for audit purposes. If there is no legal basis, obtain separate consent and store it separately.
• "Since it is a vendor model, the vendor manages the risk." Third-party AI is also subject to risk management by the adopting company.
• "Governance is an adoption approval meeting." Inventories, responsible persons, review triggers, and suspension authorities must function across the entire life cycle.

Summary at a glance

Check the official docs

• NIST AI RMF — Characteristics of trustworthy AI — Read the seven characteristics, trade-offs, and the three categories of bias.
• NIST AI RMF — Core — Locate and read GOVERN 1.6, 2.1, 2.3, 6.1; MEASURE 2.11, 3.3; and MANAGE 2.4, 3.1, 4.3.
• 29 CFR 1607.4 — Information on impact — Read the original text of paragraph (D) on the 4/5 rule of thumb and its limitations.
• Article 2 of the Equal Employment Opportunity and Work-Family Balance Assistance Act — Check the definitions of indirect discrimination and affirmative action.
• Article 4-4 of the Act on the Prohibition of Age Discrimination in Employment and Aged Employment Promotion — Check the provisions deeming discrimination based on criteria other than age as age discrimination.
• Article 33 and Article 34 of the AI Basic Act — Read about reviewing high-impact classification to verify with vendors and business operator obligations.

Self-check

5-1. Write the three categories of bias described by the NIST AI RMF, and state in one sentence the implication for audit practice of the statement "a system that has reduced harmful bias is not necessarily fair."

5-2. There is a group where 450 out of 1,500 applicants passed, and a group where 120 out of 600 applicants passed. Calculate the selection rate ratio and interpret it in light of the 4/5 rule of thumb.

5-3. Explain each of the two directions (smaller differences, greater differences) cited by 29 CFR 1607.4(D) as exceptions to the 4/5 rule of thumb.

5-4. State why stage-by-stage analysis is necessary for an organization that has calculated only final pass rates by group, and state two actions that can be chosen when stage-by-stage analysis narrows down the cause to the human evaluation stage.

5-5. State two reasons why setting different cutoff scores for each group is not a solution.

5-6. Write the three triggers for a fairness audit, state which trigger a vendor's model update falls under, and specify the materials that should be requested from the vendor at that time.

5-7. (Design) Design a one-page operating rules document for a single recruitment AI tool covering the six elements (responsible person, change control, suspension authority, audit schedule, appeals process, record retention), and link each element to one subcategory of the NIST AI RMF.

Hands-on tasks