Appendix B. Self-check answers and explanations

Chapter 1

1-1. It means first determining the question you intend to answer with the inventory, and including in the inventory only the items necessary to answer that question. To assess personal information risk, "types of data handled and whether personal information or sensitive information is included" and "vendor and entrustment relationships (who the data goes to)" are required. Name, department, and cost alone cannot answer this question.

1-2. Connections by employees working from home do not go through the company proxy, so they do not appear in proxy logs. As supplementary sources, there are corporate card and expense reimbursement statements if individuals paid personally and submitted expenses, and SSO application lists and sign-in logs if they registered using corporate accounts. Built-in features of existing SaaS are verified separately through administrator settings and vendor notices.

1-3. The scale (number of uses, departments, duration) is identified through independent evidence, the magnitude of risk (what data left the organization) through sampling, and the root cause (whether the approval process keeps up with demand) through interviews and procedural reviews. Internal audit does not hand over user lists for disciplinary purposes. Whether to take disciplinary action is a management decision; if internal audit takes it on, it assumes management's responsibility, and it does not help resolve the root cause.

1-4. It is the principle that when multiple signals are gathered in a preliminary survey, the most prominent signal determines where to concentrate time and procedures. If a model operated by an external developer has not changed for two years and human review is functioning, but the contract lacks audit rights and change notifications, the audit's focus becomes contract and vendor management rather than change management. Even for the same model, if recent replacements were frequent, the focus shifts to change management.

1-5. Yes, they can be used. The Yellow Book states that criteria can include standards, established business practices, and benchmarks (GAO 8.124), and the IIA Standards state that the most relevant criteria should be identified and used (IIA 13.4). The report states the reason for choosing those criteria and clarifies their nature as not being legal obligations.

1-6. The request list includes: the key contents of risk management measures and explanation measures; user protection measures; screens displaying the names and contact information of managers and supervisors; the system for retaining evidence of accountability fulfillment for five years and actual retained documents (Article 27(2) of the Enforcement Decree); and, in the case of a deployer, materials received from the developer and evidence that no significant functional changes occurred.

1-7. Example design: ① Because the chief audit executive temporarily assumed responsibility for that committee, assurance over that area is provided by an independent third party (e.g., an external assurance provider reporting independently to the board) for the period served and the subsequent 12 months. ② The chief audit executive develops a plan to transfer the committee secretary responsibilities to management, and internal audit staff who participated in committee operations are removed from related engagements. ③ Document this objectivity issue and the actions taken in the audit plan and share them with the board. The first parts of ① and ② are requirements of IIA 7.1 (sharing the same intent as IIA 2.2, which states that assurance for functions for which the chief audit executive had responsibility requires external independent oversight), and the rest are methods for implementing those requirements.

Chapter 2

2-1. Design effectiveness examines whether a policy establishing that "changes to decision criteria are also subject to approval" can prevent unreviewed changes if followed as intended. Operating effectiveness examines whether all actual changes to decision criteria over the past year underwent approval, and whether the approvers possessed the appropriate authority and competency.

2-2. It is an operating deficiency. AS 2201 .A3 defines an operating deficiency as a case where the person performing the control does not possess the necessary authority or competence to perform the control effectively. However, if the procedure itself that permits assignment without training is the problem, it is addressed together in the root cause analysis.

2-3. The order is inquiry, observation, inspection of documentation, and reperformance (AS 2201 .50). Inquiry is merely the statement of the person in charge and is not independent evidence demonstrating that the control was actually performed throughout the period; therefore, inquiry alone cannot serve as the basis for a conclusion.

2-4. Authority, information, time, and competence. Without authority, AI results are automatically applied even when they conflict with human opinion; without information, many cases are finalized without opening source materials; without time, the processing time per case is too short to read source materials; and without competence, override rationales do not align with criteria or overrides rarely occur.

2-5. Model retraining or replacement, adjustment of decision threshold scores, instruction modification, knowledge document updates, and vendor foundation model replacement. The two items easily missed in traditional change management are instruction modification and knowledge document updates. This is because outputs change even when code remains unchanged.

2-6. Type (whether it addressed only design or tested operating effectiveness), scope (which control objectives were addressed), period (whether it covers the audit period and how much time has elapsed), and complementary user entity controls. The aspect that most frequently clashes with AI-specific risks is scope, and if the service changes models frequently, the period also frequently becomes an issue. This is because reports commonly address only access control and availability, and do not address the use of customer data for model training or unannounced model replacements.

2-7. Example: ① Is there an inventory of tools and permissions that the agent can use, and is it reviewed regularly? ② Are those permissions the minimum necessary for the task (e.g., refund limits, recipient restrictions)? ③ Are irreversible operations (e.g., executing refunds, mass distributions) gated by human approval? ④ Are the agent's action logs recorded in a location that cannot be modified by the agent or operators?

Chapter 3

3-1. Relevance, validity, and reliability. Satisfaction may reflect whether an answer was friendly or plausible, but it does not represent accuracy; therefore, it is weak in validity, which is the degree to which a measure actually represents what it intends to measure.

3-2. Testing the accuracy and completeness of the information directly, and testing controls over the process of producing that information (AS 1105 .10). Direct testing is appropriate for materials produced manually on a one-off basis for this audit, whereas testing controls is appropriate for materials generated automatically and repeatedly under effective controls.

3-3. The results of selecting specific items cannot be projected to the population as a whole (AS 1105 .27). Disclose the selection criteria and the number of examined items in the report, and limit the conclusion to that scope, such as "among the selected high-risk items, X items" (GAO 9.21).

3-4. Tolerable rate of deviation, expected rate of deviation, and the risk of overreliance on controls (AS 2315 .38). Population size has almost no effect on sample size unless the population is very small.

3-5. Sampling risk—that is, the possibility that even though the sample deviation rate is lower than the tolerable deviation rate, the actual population deviation rate exceeds the tolerable rate—and qualitative aspects of deviations—that is, whether the deviation is a simple mistake or intentional circumvention, along with its cause (AS 2315 .41–.42). If necessary, increase the sample size or add other evidence.

3-6. Because deterministic scoring models produce identical results for identical inputs, they can be recalculated using the approved version and compared against records. Because generative models can produce different outputs even for identical inputs, past records cannot be evaluated by re-running the model; therefore, reperformance assesses whether decisions met criteria using the recorded inputs and outputs. The necessary logs are original input text, original output text, and version information for the model, instructions, and knowledge documents.

3-7. Example: ① Document in writing the criteria for "suspected fraud" tailored to the audit objective. ② After confirming the tool's data processing conditions, measure separately the tool's false positive rate and false negative rate using a representative validation sample evaluated directly by the auditor according to the criteria. ③ If both rates are within acceptable tolerances, classify the entire population, verify cases flagged by the tool against the original text, and sample from "Not Applicable" cases to check whether any cases were missed. ④ Record the instructions, model version, execution date and time, number of input items, and validation results in the workpapers.

Chapter 4

4-1. TP 150, FP 150, FN 50. Recall = 150 ÷ 200 = 75%, Precision = 150 ÷ 300 = 50%, Majority class baseline—that is, accuracy when classifying all transactions as normal = 199,800 ÷ 200,000 = 99.9%.

4-2. Validation confirms through objective evidence that the requirements for a specific intended use have been fulfilled—that is, asking "does it perform properly for this intended use?" Reliability is the ability to function as required without failure under given conditions for a given period of time—that is, asking "does it function as required without failure under specified conditions of use for a specified duration?" Whether performance is maintained across diverse situations is a question of robustness.

4-3. Separation (the problem that arises from a lack of separation is called data leakage). If rows are randomly shuffled and split, records from the same customer or the same account enter both training and evaluation, allowing the model to memorize individual characteristics and achieve scores higher than in reality. Re-evaluate by splitting on a customer or account basis.

4-4. Systemic bias, computational and statistical bias, and human-cognitive bias. Because NIST explains that all three can occur without prejudice or discriminatory intent, the absence of intent does not constitute evidence of the absence of bias.

4-5. The ground truth itself is past human judgment, so the biases embedded in that judgment are contained within the ground truth. High agreement with past decisions may simply mean that it closely mimics those very biases. The labels must be examined first against independent criteria, such as actual repayment outcomes.

4-6. For adjusting decision threshold scores, a before-and-after comparison of metrics by error type and by demographic group is appropriate; for partial updates of knowledge documents, a regression evaluation focused on questions grounded in those documents is appropriate; and for foundation model replacement, a comprehensive re-evaluation and approval prior to deployment is appropriate.

4-7. Example: The evaluation question set should reflect the distribution of actual inquiry types while sufficiently including high-consequence error categories such as eligibility requirements, income criteria, and application deadlines. Scoring criteria should be established by cross-referencing the original announcement and guidance texts, and scored independently by two evaluators to check the agreement rate. For high-risk categories, report accuracy separately from the overall average. During operations, monitor the sample accuracy of high-risk categories, the number of appeals, and agent modification rates on a monthly basis, and respond if thresholds are exceeded. Because modification rates may appear artificially low due to automation bias, independently review a sample of responses sent without modification alongside them.

Chapter 5

5-1. Criteria, condition, cause, and effect. The Yellow Book states that the four elements should be developed to the extent necessary to achieve the audit objectives, and explains that if the objective is solely to determine the condition, developing only the condition is sufficient (GAO 8.116, 8.128).

5-2. If you stop at an individual employee's action, the recommendation becomes training or disciplinary action, and the same issue repeats with other employees. You must follow the evidence further to see why that employee did not reflect it. Example of a better cause statement: "Because no responsible person or deadline was designated to reflect revisions of criteria into knowledge documents, the revision was not communicated to the operations department."

5-3. The severity of a deficiency does not depend on whether actual harm occurred, but on the reasonable possibility that controls will fail to prevent or detect an error and on its potential magnitude (AS 2201 .63–.64). Therefore, it is determined by applying the company's rating criteria to that possibility and magnitude.

5-4. It must be directed to the party with the authority to act, and must be specific, actionable, cost-effective, and measurable. "Strengthen the AI management system" lacks an addressee, specific actions, and measurement methods, and shows no connection to the cause.

5-5. ① Since the comment conflicts with the finding, evaluate its validity and supporting evidence. ② If the comment is supported by sufficient and appropriate evidence, revise the report. ③ Otherwise, maintain the finding and include the responsible official's written comments (or a summary thereof) in the report along with the reasons for disagreement. Agreement from the audited organization is not a condition for issuing the report.

5-6. (a) A manual review performed quarterly can be closed only after it has operated several times and performance records during the period are tested on a sample basis. (b) A system configuration change can be closed after directly verifying the configuration, observing its operation with test inputs, and confirming that the configuration was not changed thereafter under effective change management.

5-7. Example: ① Obtain written documentation of management's decision, its rationale, and the decision-maker. ② The head of audit determines whether the accepted risk is within the organization's risk appetite and risk tolerance. ③ If within the tolerance, record the acceptance decision and its rationale in the tracking system and conclude follow-up on that finding. ④ If it exceeds the tolerance, discuss it with executive management, and if not resolved, inform the board. Because the responsibility for risk acceptance and action rests with management, the audit team does not design controls on their behalf.