Chapter 5. Reporting and Follow-up

Learning objectives

• Write findings with supporting evidence across the four elements of criteria, condition, cause, and effect.
• Confirm causes through evidence, and select recommendations that target those causes the organization can address.
• Assign risk ratings and remediation deadlines to findings in accordance with the company's established methodology.
• Appropriately incorporate management's views, audit scope and limitations, and sample projectability into the report.
• Verify the completion of follow-up actions with evidence appropriate to the action type, and report risk acceptance to the appropriate parties.

Introduction

An audit team at a city hall identified an issue with an AI civil complaint guidance chatbot. The first sentence of the draft report read: "As there are deficiencies in chatbot operations, strengthen management." The responsible department nodded upon reading this sentence, but nothing had changed even three months later. This was because it lacked what was deficient, why it was deficient, and consequently who needed to do what by when.

A good finding is written so that the reader understands the next action. It shows the difference between the state that should exist (criteria) and the actual state (condition), and supports with evidence the factor that produced that difference (cause) and the outcome that resulted from that difference (effect). The cause must be pinpointed accurately for recommendations to be effective, and the effect must be estimated to determine the urgency of action. Furthermore, an audit does not end on the day the report is issued. It ends only after verifying that the recommendations actually addressed the cause. Because findings in AI audits often directly impact citizens or customers, this follow-up stage of verifying recommendation implementation carries even greater weight.

This chapter first covers the four elements of a finding, identifying causes, estimating effect, and risk ratings. Next, it examines how to incorporate recommendations, management's views, expressions of conclusions, and scope and limitations into the report, and finally summarizes methods for follow-up verification and responding when management accepts risk.

Key concepts

The four elements of a finding

The Yellow Book requires auditors to plan and perform procedures to develop the criteria, condition, cause, and effect to the extent necessary to satisfy the audit objectives when they identify a finding (GAO 8.116). As noted in the preface, the Yellow Book represents U.S. government auditing standards, and this book borrows its principles regarding findings and reporting. If there are domestic audit regulations that your institution must follow, those take precedence. IIA standards also state that findings begin with comparing criteria and condition, exploring root cause and effect, and evaluating their significance (IIA 14.3).

Not all four elements must always be present. If the audit objective is solely to determine the condition, developing only the condition is permissible (GAO 8.128). However, for a finding that leads to improvement, cause and effect must not be omitted.

Identifying causes

The Yellow Book defines cause as the factor that produced the difference between criteria and condition, and cites poorly designed policies, procedures, and criteria, inconsistent, incomplete, or incorrect implementation, and factors beyond management's control as common causes (GAO 8.126). In addition, when internal control is significant to the audit objectives, it requires considering internal control deficiencies when identifying cause (8.117), and explains that the root cause of a finding involving violations of laws or regulations can be a defect in control design (8.129).

The simplest method for identifying root cause is repeatedly asking "why." IIA standards state to collaborate with management when identifying causes. This is because the people performing the work understand the differences best.

1. Why did the chatbot provide the outdated baseline amount? → The knowledge document was not updated.
2. Why was it not updated? → The employee in charge was unaware of the revision to the baseline.
3. Why was the employee unaware? → No responsible party had been designated to reflect baseline revisions in knowledge documents.

If one stops at "employee error," the recommendation becomes training or disciplinary action, and the same incident repeats with another employee. While the cause should be stated as the actual factor verified through evidence, one must trace back to the roles, procedures, system configurations, and contractual terms that made that error possible. If the cause is an external factor beyond management's control, state it as such, and direct the recommendation toward responses the organization can undertake.

Estimating and expressing effect

Effect demonstrates why action is needed (GAO 8.127). In an AI audit, effect is typically described under three categories:

• Outcomes that have already occurred: Number of incorrectly guided inquiries, erroneously rejected applications, blocked legitimate transactions.
• Potential outcomes: The scale of potential harm that could occur in the future if controls remain unchanged.
• Affected individuals: Who among customers, citizens, or employees, and how many.

The Yellow Book requires auditors to describe identified instances in relation to the population or the number of items reviewed and, where feasible, quantify them in monetary or other terms so readers can assess the nature and magnitude of the finding. If results cannot be projected to the population, conclusions must be appropriately limited (GAO 9.21). If 6 errors were found out of 60 cases selected based on risk indicators alone, one cannot write "10% of the total was erroneous"; rather, one must write "6 out of 60 selected high-risk cases."

Risk rating and prioritization

IIA standards require evaluating the significance of each finding based on risk likelihood and impact, and prioritizing them according to the methodology established by the chief audit executive (IIA 14.3). The rating is determined by the company's methodology, and the auditor applies that methodology consistently to the facts.

A common mistake when assigning ratings is lowering the rating simply because actual harm has not yet materialized. Regarding controls over financial reporting, AS 2201 explains that the severity of a deficiency does not depend on whether a misstatement actually occurred, but rather on whether there is a reasonable possibility that the controls will fail to prevent or detect it and on the magnitude of the potential misstatement (AS 2201 .63–.64). While this standard applies to financial reporting, this principle can also be borrowed in AI auditing to support the judgment not to downgrade risk solely based on the presence or absence of actual harm. If price lists change annually but there is no procedure to update the price list referenced by the AI, the reasonable possibility of affecting customer finances remains intact, even if no harm occurred this quarter because prices did not change.

Even for the same type of AI error, the rating differs depending on where the output is used. An omission in an internal meeting minutes summary is an issue of operational efficiency; an error in an aggregation used by management to set budgets is an issue of decision-making; and an error in an amount quoted to customers is an issue of customer rights.

Writing recommendations

The Yellow Book requires recommendations to follow logically from the findings and conclusions, to be directed at resolving the cause of the identified deficiencies, and to clearly state the recommended actions (GAO 9.23). Effective recommendations are directed to parties with the authority to act, and are specific, practical, cost-effective, and measurable (9.28). IIA standards also require recommendations or action plans to address the difference between criteria and condition, reduce risk to an acceptable level, and address root causes (IIA 14.4).

It is also advisable to include separate actions in recommendations for individuals who have already suffered harm. Correcting the cause does not automatically rectify the 37 erroneously rejected applications.

Obtaining management comments

The Yellow Book requires auditors to obtain the views of responsible officials on the findings, conclusions, recommendations, and planned corrective actions, and to include them in the report (GAO 9.50–9.51). If the views disagree with the findings, the auditor must evaluate the validity of those views, explain in the report the reasons for disagreeing, and modify the report if the views are valid and supported by sufficient, appropriate evidence (9.52). If the audited entity refuses to provide views or fails to provide them within a reasonable period, the auditor may issue the report without them and state that fact in the report (9.53).

Two principles follow from this. Agreement from the audited entity is not a prerequisite for issuing the report. Furthermore, findings must not be omitted simply because of disagreement, nor should adverse views be concealed. It is also worth remembering that documents created after audit fieldwork has ended cannot substantiate conditions during the preceding period. IIA standards require that when auditors and management disagree regarding recommendations or action plans, both sides state their positions and reasons and follow an established process to resolve the disagreement (IIA 14.4).

Expressing conclusions and assurance

A conclusion is not a summary of findings, but a logical inference drawn from them, and its strength depends on the persuasiveness of the evidence and the soundness of the logic (GAO 9.27). IIA standards require audit conclusions to summarize results in light of audit objectives, and conclusions in assurance engagements to include an evaluation of whether the governance, risk management, and controls of the audited activity are effective—including recognition of that fact when they are effective (IIA 14.5).

In conclusions for AI audits, phrasing that broadens the scope of assurance requires special caution. Rewording "Seven AI systems were audited and no issues were found" into "The company's AI management complies with ISO/IEC 42001" reads as though assurance is provided even for unaudited systems and procedures. Because ISO/IEC 42001 specifies requirements for the organization's entire AI management system rather than individual systems, an audit of seven systems cannot speak for the whole. Furthermore, ISO does not perform certifications; certification is performed by external certification bodies. While internal audit may evaluate compliance with standard requirements within the audited scope, its conclusions must not be read as certification. Internal audit conclusions must be written strictly within the audited scope, criteria, and period.

Disclosing scope, methodology, and limitations

The Yellow Book requires reports to state the scope and limitations of the work performed so that readers do not misunderstand, and to report significant constraints such as information limitations or denial of access to records (GAO 9.12). It requires describing the relationship between the population and the items tested, the period covered, and the types and sources of evidence (9.13), as well as explaining techniques for evidence collection and analysis, criteria used, and—when sample results significantly support conclusions—the sample design, reasons for selection, and whether results can be projected to the population (9.14). If there are significant limitations or uncertainties in the evidence, they must be disclosed alongside the conclusions (9.20), and any significant facts that, if not disclosed, could mislead readers must also be disclosed (9.22). If internal control is significant to the audit objectives, the scope of work on internal control and any significant deficiencies must be included in the report (9.29).

In AI audits, the following are frequently documented in the methodology: periods that could not be verified due to log retention periods (this limitation may separately constitute a finding as a deficiency in evidence retention controls); the AI classification tools used by the audit team and their validation results; external experts who participated and their scope of work; and systems excluded from the audit scope along with the reasons for exclusion.

Principles of follow-up verification

IIA standards require internal auditors to verify whether management has implemented recommendations or action plans according to an established methodology, which includes inquiring into progress, performing risk-based follow-up evaluations, and updating status in tracking systems. The scope of procedures is determined considering the significance of the findings (IIA 15.2). The Yellow Book also requires auditors, when planning a subsequent audit, to verify the status of previous recommendations and use those results in risk assessment and audit procedures (GAO 8.30).

Management's report of "Action Completed" is the starting point of follow-up verification. Completion reports, revised procedures, and official notices designating personnel demonstrate only the design effectiveness of controls, not evidence of operating effectiveness showing that the controls operated over a period of time to resolve the cause.

Closing evidence by action type

The nature of the action dictates what evidence is required for closure.

When action is delayed, IIA standards require obtaining management's explanation, documenting it, and discussing it with the chief audit executive. The chief audit executive determines whether management has accepted risk beyond the risk tolerance level through delays or non-implementation (IIA 15.2).

Risk acceptance and reporting

The NIST AI RMF identifies mitigation, transfer, avoidance, and acceptance as responses to high-priority AI risks, and states to develop, plan, and document those responses (MANAGE 1.3). Accepting risk can be a legitimate choice, and that decision rests with management.

However, if the accepted risk exceeds the organization's risk appetite or tolerance, the situation changes. IIA standards require that if the chief audit executive determines that management has accepted risk at such a level, they must discuss it with senior management, and if not resolved, report it to the board (typically the audit committee in organizations with an audit committee). Resolving the risk is not the responsibility of the chief audit executive (IIA 11.5). The role of audit is to make that decision visible to the appropriate authority. If the risk is within tolerance, the auditor may record the acceptance decision, the rationale, and the decision-maker, and conclude tracking.

Going deeper

The impact of AI findings reaches people

The impact of a traditional internal control audit is often expressed in monetary amounts. The impact of an AI audit often reaches people directly. Examples include loan applications that were erroneously rejected, welfare benefits that people could not apply for because they were informed based on outdated criteria, and remittances blocked after being mistaken for fraud. When estimating these impacts, three numbers are counted separately: the scope of people who might have been affected (everyone who received the applicable guidance during that period), people confirmed to have actually been affected (verified through a sample or a complete analysis), and people who can still be redressed (people whose application deadlines remain).

The last number is especially important. Recommendations that remedy the cause prevent future harm, but people who have already been affected can be redressed only if separate measures are taken. Auditors verify whether records exist to locate the affected individuals and write remediation measures as a recommendation separate from addressing the root cause. The NIST AI RMF also instructs informing relevant actors, including affected individuals, of incidents and errors, and following and documenting tracking, response, and recovery procedures (MANAGE 4.3). If records do not remain and individuals cannot be identified, that fact itself becomes a finding regarding log retention controls.

When opinions differ with management

In AI audits, disagreements with management are frequent. Objections such as "A human makes the final decision, so oversight is sufficient," "The vendor warranted it, so there is no problem with the data," and "There has been no damage yet, so please lower the severity rating" overlap exactly with the misconceptions covered in earlier chapters of this book.

The principle an auditor must uphold at this time is evidence. If management's objection is supported by new evidence, the report is revised; otherwise, the objection is included in the report along with the reasons for disagreement. Ratings are assigned according to the methodology defined by the organization and are not a subject of negotiation. However, auditors must also re-examine whether their own evidence is sufficient. If the objection points to facts the auditor missed, revising the report is not a defeat, but a process of creating a fair report. This is why the Yellow Book explains that obtaining views from responsible officials by sending them a draft helps produce a fair, complete, and objective report (GAO 9.54).

At work

Starlight City: Rewriting a one-sentence finding into four elements

The Audit Office of Starlight City audited a system where AI classified citizen proposals and assigned them to responsible departments. The finding in the initial draft was a single sentence: "The accuracy of AI proposal classification is low, requiring improvement." The audit director had this sentence rewritten into the four elements.

They began by identifying the criteria. The city's proposal processing guidelines stated that "Proposals shall be assigned to the responsible department within three days of receipt," and the system deployment plan set a target of "Misassigned proposals at 5% or less." The condition was that among 200 proposals randomly sampled by the audit team, 31 (15.5%) were misassigned, and 22 of those took an average of nine days to be reassigned. Because this was a random sample, the audit team decided to report this rate as an estimate for the entire 6,400 proposals from the previous quarter, along with the magnitude of sampling risk such as confidence intervals.

Finding the cause took time. At first, "Insufficient performance of the AI model" seemed to be the cause. The audit team repeatedly asked "why." Among the 31 misassigned cases, 24 were related to departments whose names and duties had changed during the organizational restructuring in March. The system's departmental taxonomy remained as it was before the reorganization, and no department had been assigned the responsibility to reflect reorganizations in the system. The cause was not the model, but the absence of a procedure to reflect organizational restructuring in the classification table.

The effect was documented in three parts: the estimated scale of misassignments, the processing delays incurred while reassigning proposals, and the fact that delay notifications were not sent to citizens for proposals that exceeded the processing deadline. The recommendation to the Information and Communications Division was written as: "Establish responsibility and deadlines for updating the classification table during organizational restructuring, and report the misassignment rate to the next quarter following the update." Some suggested recommending model retraining, but it was discarded because it did not align with the cause. The omission of citizen notices was added as a separate recommendation to the department handling civil petitions.

Sweet Rain Energy: Three different verifications of 'Action Completed'

At city gas utility Sweet Rain Energy, AI detected anomalies in automated meter reading values to select targets for on-site inspections. Management reported "Action Completed" for all three findings from the previous year's audit. The audit team did not verify the three cases in the same manner.

The first finding was: "Field inspectors batch-close targets selected by AI without verifying supporting evidence." Management removed the batch-closure function from the inspection app and modified it so that the close button was enabled only after inspection photos were uploaded. The audit team directly verified the app settings and, using a test account that left no records in production data, confirmed that closure was impossible without photos. Because general IT controls over program changes, access, and operations for the app had been evaluated as effective that year and were continuously tested, the audit team closed the finding after confirming the settings and reviewing change logs showing no subsequent alterations.

The second finding was: "No one verifies whether there are actual anomalies among meters excluded by AI from inspection targets." Management established a procedure for the quality team to conduct monthly on-site sample checks of 30 excluded meters, which was performed for the first time the previous month. The audit team reviewed the record of the initial execution, but judged that one month of evidence was insufficient to conclude continuous operation, and scheduled sample testing after three months of operation. The status remained as "Action in Progress (Operational verification scheduled)".

The third finding was: "There are no revalidation criteria for the meter reading anomaly detection model." Management stated that they would maintain the current state without establishing a revalidation framework, citing costs. The company's risk management standards defined "missed inspections related to gas safety" as an unacceptable risk. The chief audit executive determined that this decision constituted risk acceptance exceeding tolerance limits and discussed it with the CEO; when the decision did not change, it was reported to the Audit Committee of the Board of Directors. The audit team rejected an internal suggestion to build the revalidation framework on management's behalf. That was management's responsibility, and if the audit team built it, a self-review threat would arise when evaluating their own work later.

Common mistakes

1. "The cause is an error by the person in charge." If you stop at an error, the same thing will repeat itself. Trace the evidence back to the roles, procedures, configurations, and contracts that made that error possible, and state the cause.
2. "Since there is no actual harm, lower the rating." Severity is determined by the reasonable possibility of a control failing and the magnitude of that failure (AS 2201 .64).
3. "Recommendations should be written broadly so that management can implement them according to the circumstances." Effective recommendations are directed to parties with authority and are written specifically and measurably (GAO 9.28).
4. "If management does not agree, omit the finding." Evaluate the validity of the views, and if you do not agree, state the reasons (GAO 9.52).
5. "Use the error rate of a selected sample as the overall error rate." When results cannot be projected, restrict the conclusions accordingly (GAO 9.21).
6. "Because there were no issues with the audited system, we may state that it complies with the standard." State conclusions only within the audited scope, criteria, and period. Certification is performed by certification bodies.
7. "Management reported completion, so close it out." A completion report is only the starting point. Verify with evidence appropriate to the type of action.
8. "If management accepts the risk, audit has nothing left to do." If it exceeds the risk tolerance, discuss it with senior management, and if unresolved, inform the board (IIA 11.5).

Summary at a glance

Check the official docs

• GAO — Government Auditing Standards 2024: Findings (8.116–8.130), report contents (9.12–9.29), views of responsible officials (9.50–9.54), previous audit results (8.30).
• The IIA — Global Internal Audit Standards: Standard 11.5 (Reporting Risk Acceptance), 14.3 (Evaluation of Findings), 14.4 (Recommendations and Action Plans), 14.5 (Conclusions), 15.2 (Follow-up on Implementation).
• PCAOB — AS 2201: Severity of deficiencies (.62–.64), testing period (.52), benchmarking of automated controls (.B28–.B29).
• NIST AI RMF — Core: MANAGE 1.3 (Risk response mechanisms), MANAGE 4.3 (Communicating and tracking incidents and errors).
• ISO — Certification: Explanation that ISO does not perform certification and that it is performed by certification bodies.

Self-check

5-1. State the four elements of a finding, and answer whether the remaining elements may be omitted when the audit objective is limited to determining the condition, in light of the Yellow Book.

5-2. A draft states, "The assigned employee did not reflect changes to criteria in the knowledge document" as the cause. Explain why this cause must be examined one step deeper, and write an example of a better cause statement.

5-3. Explain what should serve as the basis when determining the rating of a control deficiency for which no actual harm has been confirmed, in light of AS 2201.

5-4. State the conditions for an effective recommendation in accordance with Yellow Book 9.28, and point out which conditions the recommendation "Strengthen the AI management system" fails to meet.

5-5. Management submitted written comments disagreeing with a finding in the draft report. State the actions the audit team must take in sequence, according to Yellow Book 9.50–9.53.

5-6. When an action for which management reported "Action Completed" is (a) a new manual inspection performed quarterly and (b) a system configuration change, state the evidence required for closure in each case.

5-7. (Design) Management decided not to implement a recommendation citing cost. Design step by step the procedures that the chief audit executive should follow in accordance with IIA Standards 11.5 and 15.2. Address separately cases where the risk is within the tolerance limit and cases where it exceeds it.

Hands-on tasks