Chapter 2. Control Design and Operating Effectiveness

Learning objectives

• Distinguish between the design effectiveness and operating effectiveness of controls to select evaluation procedures.
• Determine whether an identified issue represents a design deficiency or an operating deficiency.
• Select tests to determine whether human review controls are substantive or merely formal approvals.
• Expand the scope of access management and change management in AI systems to inspect models, decision criteria, instructions, and knowledge documents.
• Read vendor contracts and service organization reports to identify items that the company must verify separately.

Introduction

The claims team lead at an insurance company confidently told the audit team: "Every amount calculated by the AI is verified and confirmed by the person in charge. A human makes the final decision, so there is no problem." It was stated that way in the procedure document as well, and only the person in charge could click the confirm button. When the audit team opened the system logs, the person in charge had revised the AI amount in only 3 out of 1,000 cases over the past quarter, and the average processing time per case was 20 seconds.

Is this control effective? The answer is "we do not know yet." The procedure document and the confirm button are merely evidence that the control was designed that way, not evidence that it actually operated during the period. Furthermore, the low revision rate proves nothing on its own. If the AI was rarely wrong, there would have been no need to revise anything; if the AI was frequently wrong, the person in charge let errors slip through. More evidence is needed to distinguish between the two. Evaluating controls begins by separately asking "does the control exist?" and "does the control operate?" in this manner.

This chapter begins by distinguishing between the two pillars of control evaluation—design and operation—and covers how to determine where an issue arose. It then examines in depth human review controls and formal approvals, which are the most frequent points of contention in AI audits, and outlines how to inspect access management, change management, and vendor management controls tailored to the characteristics of AI systems.

Key concepts

Types of controls

A control is a mechanism that prevents a risk from leading to an adverse outcome. PCAOB AS 2201 divides controls into preventive controls, which prevent errors or fraud from occurring, and detective controls, which detect errors that have occurred in a timely manner, explaining that effective internal control frequently uses both together (AS 2201 .A8). In terms of the executing entity, there are manual controls performed by humans and automated controls performed by systems.

This distinction determines the testing method. For controls repeatedly performed by humans, samples must be drawn and examined across the entire period; fully automated controls can be verified efficiently by reviewing their configuration and its change management.

Design effectiveness and operating effectiveness

Design effectiveness examines whether a control, if operated as prescribed by persons possessing the authority and competence, satisfies the control objectives and can prevent or detect errors or fraud (AS 2201 .42). Operating effectiveness examines whether the control is actually operating as designed and whether the person performing it possesses the necessary authority and competence (AS 2201 .44).

By way of analogy, evaluating design is like checking whether a fire extinguisher of the correct type is placed in the proper location, whereas evaluating operation is like checking whether its pressure was checked and it was replaced according to the checklist over the past year. If the fire extinguisher is in the wrong place, it cannot prevent a fire no matter how diligent the inspection records are. Therefore, design evaluation comes first. The IIA Standards also explain that if controls are not adequately designed, the benefits of fully evaluating their effectiveness are unlikely to exceed the costs (IIA 4.2).

Design deficiencies and operating deficiencies

AS 2201 defines the two types of deficiencies as follows (AS 2201 .A3).

• Design deficiency: When a control necessary to achieve the control objective is missing, or an existing control is designed such that it cannot achieve the objective even if it operates as designed.
• Operating deficiency: When a properly designed control does not operate as designed, or the person performing the control does not possess the necessary authority or competence to perform it effectively.

In AI systems, this distinction is confused more often than one might think. Consider the following examples.

The determination is important because the recommendations differ. A design deficiency requires revising policies or systems, whereas an operating deficiency requires correcting execution or replacing the performer.

Testing techniques and strength of evidence

There are four techniques for testing controls: inquiry, which involves asking the person in charge; observation, which involves watching the execution process; inspection of documentation, which involves examining execution records; and reperformance, where the auditor independently reperforms the control. AS 2201 lists these four in order from typically weakest to strongest evidence obtained, and clearly states that inquiry alone cannot support a conclusion that a control is effective (AS 2201 .50). The actual strength of evidence varies depending on the nature of the control and the combination of procedures performed (.49, .51).

For design evaluation, a walkthrough combining inquiry, observation, and inspection of documentation is usually sufficient (AS 2201 .43). A walkthrough may also provide sufficient evidence of operating effectiveness depending on the risk and the procedures performed (.49), but if the risk is significant or execution must be examined broadly across the period, sample testing is required. A walkthrough is a method of tracing a single transaction from start to finish, asking the person in charge questions, and viewing actual screens and records together. For operating evaluation, samples are drawn across the entire period to perform inspection of documentation or reperformance. Testing over a longer period of time provides stronger evidence regarding control effectiveness (AS 2201 .52).

When encountering deviations

When a case where the control was not performed—that is, a deviation—appears in a sample, one does not immediately conclude that the control is ineffective. AS 2201 explains that because effective controls do not provide absolute assurance, individual controls need not operate without deviation to be considered effective (AS 2201 .48). The auditor compares the frequency of deviations against the tolerable rate of deviation (Chapter 3) and also evaluates the cause and nature of the deviations. A deviation arising from a mistake carries different weight than an intentional bypass.

Conversely, a deficiency must not be taken lightly simply because no actual damage occurred. AS 2201 explains that the severity of a deficiency does not depend on whether an error actually occurred, but on whether there is a reasonable possibility that the control will fail to prevent or detect an error and the magnitude of that error (AS 2201 .63–.64). The fact that a departed employee's account was not deactivated for two weeks but had no login records can be used in the impact assessment, but it does not erase the deviation itself.

Requirements for human review controls

The most common control in AI systems is "a human verifies and then confirms." The AI Basic Act cites human management and supervision as an obligation of business operators providing high-impact artificial intelligence (Article 34, Paragraph 1, Item 4); the NIST AI RMF calls for defining, assessing, and documenting human oversight procedures (MAP 3.5), and establishing mechanisms and responsibilities to replace or shut down AI systems that produce outcomes contrary to intent (MANAGE 2.4). The GAO framework also calls for establishing human oversight procedures to ensure accountability (Practice 3.9).

For human review to be substantive, four elements must be in place.

1. Authority: Reviewers must be able to overturn the AI results. If only opinions are recorded and the AI result is automatically applied by default, supervision is perfunctory.
2. Information: Reviewers must be able to see the source data needed for judgment and the rationale for why the AI reached that decision.
3. Time: Targets per case or queue pressure must not make actual judgment impossible.
4. Competence: Reviewers must possess the knowledge and training to make that judgment.

If the review procedure was designed from the beginning without granting the authority to overturn, the source data, or the time, it is a design deficiency. If the procedure was properly created but the person actually assigned to the review lacked the necessary authority or competence, it is an operating deficiency as defined by AS 2201.

Formal approval and automation bias

Formal approval (often referred to as a "rubber stamp") refers to a state where a review exists procedurally, but in practice, the AI result is passed through as is. At its foundation lies automation bias. NIST AI 600-1 explains that as the reliability of generative AI increases, humans may become overly reliant on the system or value its outputs more highly than other sources, and that such overreliance on automated systems can amplify other risks such as confabulation or bias.

Signals of formal approval are typically as follows:

• The rate of modification or reversal is significantly lower than the known error rate of the AI.
• The processing time per case is too short to read the source data.
• A batch approval feature exists and is actually used.
• There is a requirement to record reasons when reversing, but no records are kept when following the AI output as is.

These signals are "suspicions," not "conclusions." Experienced reviewers are no exception, so they must not be excluded from testing on the grounds of their proficiency.

Selecting tests to identify formal reviews

Which test is appropriate depends on what evidence remains, what is permitted, and what the audit team is capable of judging.

Injecting test cases is powerful, but because it can impact production systems, internal company regulations and management agreement must come first. Reperformance is meaningful only when the audit team is capable of making the same judgment; if not, qualified experts should be involved. A test that concludes with inquiry alone is not sufficient in any situation.

Access control

Access control in AI systems addresses "who can alter the results." Beyond model files and deployment configurations, instructions (prompts) for generative AI, decision criteria scores, knowledge documents used in retrieval, and external tool connection settings all alter results. Auditors verify the following:

1. Is modification access granted only to those who strictly need it (least privilege)?
2. Is the permissions list reviewed regularly? The review reconciles the permissions list against independent records, such as cross-referencing it with the list of active employees from HR records.
3. Are permissions revoked within the deadline established by policy upon departure or transfer?
4. Is a modification history maintained, and does someone review that history?

Modification history is a detective control that identifies actions after the fact. The fact that history is recorded does not replace preventive controls such as least privilege.

Change management controls

Change management in traditional IT audits focuses on program code changes. In AI systems, the unit of change is broader. Model retraining or replacement, adjustment of decision criteria scores, modification of instructions, updates to knowledge documents, and foundation model replacement by external vendors all alter outcomes. If the change management policy designates only a subset of these as requiring approval, leaving the remainder outside of control is a design deficiency.

The sequence for testing change management controls is typically as follows: identify the changes requiring approval and authorized approvers in the policy; directly extract actual deployment histories from the production system as an auditor; reconcile them one by one against approval records; and investigate the causes of any discrepancies. The auditor also checks whether the requester and approver are the same person and whether the approved content matches the deployed content.

Automated controls and benchmarking strategy

Fully automated controls, such as "the confirm button is enabled only after entering a rationale," are generally not subject to the risk of human failure (AS 2201 .B28). AS 2201 explains that if general controls over program changes, access, and operations are effective and continue to be tested, and the auditor verifies that the automated control has not changed since the baseline was established (since it was last tested), the auditor can conclude that the automated control continues to be effective (.B29). This is referred to as a baseline (benchmarking) strategy.

However, the operation of automated controls may depend on related files, tables, data, and parameters (AS 2201 .B30), and the baseline must be reestablished over time (.B33). In an AI system, decision criteria scores or reference price tables are examples of such parameters. If "the button configuration remains the same, but the reference table has changed," the control has changed.

Vendor management controls

Many AI systems run on models or services from external vendors. Even when outsourced, the risk remains with the company. The NIST AI RMF calls for policies and procedures to address third-party-related AI risks (GOVERN 6.1), contingency procedures in the event of failures of third-party data or AI systems deemed high-risk (GOVERN 6.2), regular monitoring of risks from third-party resources (MANAGE 3.1), and including pre-trained models used in development in regular monitoring (MANAGE 3.2). If processing of personal information was outsourced, the outsourcing documents and supervision of trustees under Article 26 of the Personal Information Protection Act (PIPA) are also examined.

Clauses that auditors specifically look for in AI vendor contracts are as follows:

Security certifications, outage compensation, and confidentiality clauses are also important, but one must distinguish that they do not directly address risks unique to AI, such as unannounced model replacements and performance degradation.

Reading service organization reports

Vendors often provide service organization reports prepared by independent auditors. Although PCAOB AS 2601 is a standard dealing with such reports in financial statement audits, the principles of reading these reports can be directly applied to other assurance reports provided by AI vendors. Key points to remember summarized from AS 2601 are as follows:

1. Type of report. A report that only addresses whether controls were designed and placed in operation as of a specific date is helpful for planning the audit, but does not provide evidence of operating effectiveness (AS 2601 .12, .37). It must be a report that tests operating effectiveness over a period of time to serve as such evidence.
2. Scope. The report expresses an opinion only on the stated control objectives and expresses no opinion on control objectives that were not tested (.44). A report covering only access controls and availability says nothing about whether data was used for model training.
3. Period. The shorter the testing period or the longer the time that has elapsed since the test, the less assurance the report provides (.16).
4. Complementary user organization controls. If the service organization's controls were designed under the assumption of user controls, the report identifies those controls (.31, .46). If the company does not operate those controls, the report's conclusions lose their premise.

Going deeper

The cost of meaningful human review

A design where "a human carefully reviews every case" sounds safe, but in practice it easily leads to perfunctory approval. If a reviewer who must process hundreds of cases a day is allowed only a few seconds per case, review drifts toward following the AI's results, regardless of what the procedure manual requires. If an auditor merely recommends "strengthening the review" in such situations, the same problem repeats.

A better design is to differentiate reviews according to risk. Cases involving large amounts, decisions adverse to customers, or low AI confidence are reviewed in depth by a human, while other cases are addressed through detective controls such as sample re-reviews. At this point, there are three things for the auditor to verify: whether the segmentation criteria align with risk, whether sufficient time and information are provided for cases reviewed in depth, and whether the results of sample re-reviews actually lead to improvements in criteria or models. The decision itself to reduce human review is a management risk judgment, and the auditor examines whether that decision is within the risk tolerance level and who approved it.

Agent tool permissions and human approval

AI agents directly invoke tools to send emails, place orders, and modify records. The scope of access control thus broadens from human accounts to agent permissions. As countermeasures against prompt injection (an attack where hidden instructions in user input or external documents alter the model's behavior away from its intended purpose), OWASP recommends restricting the model's access permissions to the minimum necessary for the intended task, requiring human approval for high-privilege operations, and regularly conducting penetration testing that treats the model like an untrusted user. Writing "Do not expose personal information" in the prompt instructions is merely one of those countermeasures; it cannot be considered secure on its own.

The auditor's questions shift into the same framework as human accounts: Is there an inventory of tool permissions held by the agent? Are those permissions the minimum necessary for the task? Is human approval placed on irreversible operations (fund transfers, deletions, external transmissions)? Are the agent's action logs recorded in a location that the agent itself cannot modify? The last question leads into log integrity in Chapter 3.

At work

Single Thread Insurance: What a 0.6% modification rate means

At non-life insurance company Single Thread Insurance, when AI classified travel insurance claims into "Payment" and "Review," nine handlers verified and finalized the "Payment" cases. In the operational data received by the audit team, the rate at which handlers changed the AI's determination back to "Review" was 0.6%, and the median verification time per case was 11 seconds. The head of the claims department explained, "Because the AI is accurate, there is little to correct."

The audit team first looked for a baseline to verify whether this explanation was correct. The pre-release validation report showed that approximately 4% of "Payment" determinations were not eligible for payment under the policy terms. If the AI was incorrect 4% of the time but humans corrected only 0.6%, the discrepancy was substantial. However, because the model might have been improved after release, the team used this discrepancy not as a conclusion, but as the starting point for testing.

Three options were considered for the testing method. Handler interviews could not produce a conclusion through inquiry alone, so they were relegated to a secondary procedure. The option of inserting test error cases into the queue involved entering claims into the production system that would not be processed, which required approval from the Information Protection Committee under internal regulations, and obtaining approval within the audit period was difficult. What remained was reperformance. All original claim documents were preserved, and travel insurance policy evaluations could be performed by an insurance professional on the audit team who had never been involved in the claims review of that product.

The audit team randomly sampled 60 cases from the 21,000 "Payment" cases finalized during the previous quarter and re-evaluated them according to the policy terms. Among them, two cases were not eligible for payment, yet had passed handler verification. The audit team also examined the verification screen. On the screen, only the AI determination and amount were displayed prominently, attached documents opened only after clicking a separate tab, and there was a "Confirm All" button on the screen. The audit team organized the results into two streams. They evaluated the two cases as operational deviations (the evaluation against the tolerable deviation rate was performed according to the method in Chapter 3), and recorded the screen configuration—which allowed finalization without opening documents—and the batch confirmation button as design-level causes that failed to provide reviewers with the necessary information. In the report, they cited this design problem as the cause of the finding, and the two missed cases as evidence demonstrating its impact.

Willow Foods: Is a single vendor report sufficient?

The food manufacturer Willow Foods summarized supplier contracts and flagged risk clauses using a third-party generative AI service. When the audit team requested evidence regarding vendor controls, the procurement team presented a service organization report provided by the vendor, stating, "It is an unqualified opinion, so there is nothing else to examine."

The audit team read the report across three areas. First, type. The report was one that tested the operating effectiveness of controls over a 12-month period. Second, scope. The control objectives covered access management, change management, and backup and recovery. One of the audit team's areas of interest—"Is original contract text used for vendor model training?"—was not included in any of the control objectives. Third, complementary user entity controls. The report was predicated on: "Client organizations immediately deactivate terminated employee accounts in the management console."

The audit team divided what was verified by the report and what was not into a table. While using the report as evidence for change management and access management, the team cross-checked Willow Foods' list of 34 console users against HR records to verify the complementary controls. Accounts for two terminated employees were still active, and one had left the company six weeks prior. Because whether data was used for training fell outside the scope of the report, they separately verified the contract, data processing terms, and the administrator settings screen. Although the contract contained a clause prohibiting use for training, "Data Sharing for Service Improvement" in the administrator settings was turned on.

The procurement team had relied on the term "unqualified opinion," but reading what that opinion actually pertained to revealed two tasks that the company needed to perform directly. The audit team recorded the failure to revoke terminated employee accounts as an operating deficiency, and the data-sharing configuration as a setting inconsistent with contractual terms, each as a separate finding. As the cause of the second issue, they noted the absence of a procedure to reflect and verify contractually defined data terms in administrator settings.

Common mistakes

1. "Because there is a procedure manual, the control is effective." A procedure manual is merely evidence of design. Operation throughout the period is verified through operating tests.
2. "Because a human makes the final decision, oversight is sufficient." Oversight is substantive only when the human possesses the authority, information, time, and competence to overturn the result.
3. "Because the modification rate is low, the AI is accurate." The modification rate reflects only the degree to which humans followed the AI. It gains meaning only when compared against the AI's actual error rate.
4. "If the review time is long, it is a substantive review." The time a screen remains open does not reflect the quality of judgment. Direct evidence such as reperformance is required.
5. "Interviewing multiple people can confirm operating effectiveness." Inquiries alone cannot lead to a conclusion, and increasing the quantity of the same type of evidence does not compensate for evidentiary quality.
6. "If even a single deviation occurs, the control has failed." Judgment is made by evaluating the frequency and causes of deviations alongside tolerable thresholds.
7. "Because modification logs are kept, broad permissions are acceptable." Logs are detective controls. They cannot substitute for preventive controls such as least privilege.
8. "If the vendor report is unqualified, vendor-related risks are eliminated." One must read the report's type, scope, period, and complementary user entity controls to understand what has actually been verified.

Summary at a glance

Check the official docs

• PCAOB — AS 2201: Design and operating effectiveness (.42–.52), definition of deficiency (.A3), severity (.62–.64), benchmarking of automated controls (.B28–.B33).
• PCAOB — AS 2601: Type, scope, period, and complementary user entity controls in service organization reports.
• NIST AI RMF — Core: MAP 3.5 (human oversight), MANAGE 2.4 (fallback and shutdown mechanisms), GOVERN 6.1 and 6.2, MANAGE 3.1 and 3.2 (third parties).
• NIST AI 600-1: Human-AI Configuration and explanation of automation bias.
• GAO-21-519SP: Practice 3.9 (human oversight) and illustrative audit procedures.
• OWASP GenAI — LLM01 Prompt Injection: Countermeasures such as least privilege, human approval, and penetration testing.
• The IIA — Global Internal Audit Standards: Cost-benefit considerations in Standard 4.2.

Self-check

2-1. Explain the difference between design effectiveness and operating effectiveness in two sentences without using the fire extinguisher metaphor, using approval of changes to AI decision criteria as an example.

2-2. Does a situation where the reviewer is replaced with an untrained new employee fall under a design deficiency or an operating deficiency? Answer based on the definitions in AS 2201.

2-3. List the four control testing techniques starting with the weakest evidence, and explain why testing limited to inquiry alone cannot serve as the basis for a conclusion.

2-4. State the four elements required for a human review to be substantive, and link each element to a sign of nominal approval that appears when that element is missing.

2-5. List five types of changes in an AI system that must be subject to change management, and select two among them that are easy to overlook in traditional IT change management.

2-6. State the four items to verify upon receiving a service organization report, and select the item that most frequently conflicts with AI-specific risks, along with the reason.

2-7. (Design) Formulate four access control questions for an auditor to check regarding an AI agent that can send emails to customers and execute refunds.

Hands-on tasks