Chapter 3. Evidence Collection, Testing, and Sampling

Learning objectives

• Distinguish between the sufficiency and appropriateness of audit evidence, and explain the conditions that enhance reliability.
• Select procedures to verify before using company-produced information and AI-generated materials as evidence.
• Select an appropriate method among 100% examination, selecting specific items, random sampling, and stratified sampling in accordance with audit objectives.
• Explain the factors determining sample size, and evaluate sample results against tolerable criteria.
• Design procedures to reperform control operations using AI output logs, and reflect the reproducibility limitations of generative AI.

Introduction

An audit team at a credit card company was testing human review controls for AI credit limit reviews. The department in charge stated that "all review records exist" and sent an Excel file. The file contained 32,000 organized rows of review cases over the past year, and every row had a reviewer name and 'Verified' written on it. A junior auditor wrote in the workpapers, "Since there are more than 30,000 cases, the evidence is sufficient."

The senior auditor asked three questions: Who prepared this file and how? Does it match the number of review cases recorded in the system? And is the text 'Verified' evidence that the reviewer actually exercised judgment? The 32,000 rows indicate quantity, not quality. A file compiled by the audited department is less reliable than data obtained directly by the auditor, and rows may have been omitted or altered during the compilation process.

This chapter covers evidence that supports audit conclusions. First, it summarizes the sufficiency and appropriateness of evidence, as well as general principles of reliability, and explores the conditions for using company-produced information as evidence. Next, it addresses the three methods of selecting items and the design, size, and evaluation of samples. Finally, it examines how to handle newly emerging evidence in the AI era—namely, logs, materials produced by generative AI, and the results of AI tools used directly by auditors.

Key concepts

Sufficiency and Appropriateness

The auditor must obtain sufficient appropriate audit evidence to provide a reasonable basis for supporting the conclusions (AS 1105 .04; GAO 8.90). Sufficiency is the measure of the quantity of audit evidence, and appropriateness is the measure of the quality of audit evidence, namely its relevance and reliability (AS 1105 .05–.06). The greater the risk, the more evidence is needed, and the higher the quality of evidence, the less additional corroborating evidence may be needed. However, merely obtaining more evidence of the same type cannot compensate for low-quality evidence (AS 1105 .05).

The Yellow Book also summarizes the same principle into two presumptions: the higher the audit risk, the more and better evidence is needed, and when stronger evidence is present, less evidence may be sufficient (GAO 8.101). Furthermore, it clearly states that a large volume of evidence does not compensate for a lack of relevance, validity, or reliability (GAO 8.102). The IIA standards followed by internal auditors also require that information used for analysis and evaluation be relevant, reliable, and sufficient, explaining that information obtained directly by the auditor, from an independent source, or corroborated by other information possesses stronger reliability (IIA 14.1).

Relevance, Validity, and Reliability

The Yellow Book explains appropriateness in terms of three components (GAO 8.102).

• Relevance: The extent to which evidence has a logical relationship with, and importance to, the issue being addressed. A record showing that a sample reviewer clicked 'Verified' is relevant to whether "a review was performed," but its relevance to whether "the review filtered out errors" is weak.
• Validity: The extent to which evidence actually represents what it is intended to measure. Customer satisfaction is not valid evidence for measuring the accuracy of responses.
• Reliability: The consistency of results when measured or tested, and the extent to which the results are verifiable. It can be enhanced through corroborating evidence, statistical testing, and confirmation from other sources.

General Principles of Reliability

AS 1105 and the Yellow Book summarize general principles for assessing the reliability of evidence in similar ways (AS 1105 .08; GAO 8.104).

These principles hold true "in general" and serve as a starting point for judgment. Even with an independent external source, reliability decreases if the source lacks knowledge of the subject matter or attaches qualifying conditions (AS 1105 .08 note).

Procedures for Obtaining Evidence

AS 1105 describes seven procedures for obtaining evidence (AS 1105 .15–.21): inspection (examining records or assets), observation (looking at a process being performed by others; limited to the point in time of observation, and observation itself may alter behavior), inquiry (seeking information from knowledgeable persons; inquiry alone does not provide sufficient basis for a conclusion), external confirmation (a direct written response to the auditor from a third party), recalculation (checking mathematical accuracy), reperformance (the auditor's independent execution of procedures or controls that were originally performed by the company), and analytical procedures (evaluating plausible relationships among data and investigating unexpected differences).

In AI audits, recalculation and reperformance are used particularly frequently. For a deterministic scoring model that always yields the same result for the same input, it can be rerun with the same input; for judgments made by humans, they can be re-evaluated against the same criteria.

Information Produced by the Company

Most evidence in AI audits is information produced by company systems. Performance reports, processing logs, and dashboards all fall into this category. AS 1105 requires that when using information produced by the company as evidence, the auditor test the accuracy and completeness of the information, or test controls over its accuracy and completeness (including IT general controls and automated controls, where applicable), and evaluate whether the information is sufficiently precise and detailed for audit purposes (AS 1105 .10).

The Yellow Book applies this requirement even to data extracted directly by the auditor. The sufficiency and appropriateness of computer-processed information are evaluated whether the information is provided to the auditor or extracted directly by the auditor, and that evaluation includes reviewing completeness and accuracy in light of the intended purpose (GAO 8.98). Extracting data directly from a system does not eliminate the need to verify whether extraction criteria were correct and whether any records were omitted.

Which method to use depends on the situation.

Three Methods of Selecting Items

There are three methods of selecting items for testing (AS 1105 .22):

1. Selecting all items (100% examination): Examining the entire population. It is appropriate for a population consisting of a small number of large-value items, significant risks where other methods cannot provide sufficient evidence, and when audit procedures can be automated and applied effectively to the entire population (.24).
2. Selecting specific items: Selecting key items, suspicious items, items with a history of error, or items exceeding a certain monetary threshold (.25). The results of items selected in this way cannot be projected to the entire population (.27). The selection criteria and the number of examined cases must be disclosed in the report, and conclusions must be limited to that scope (GAO 9.21).
3. Audit sampling: Applying procedures to a portion of the population to evaluate characteristics of the population (.28).

Because AI systems preserve most processing logs as structured data, full population analysis becomes a natural choice for attributes determinable by rules, such as "whether a reviewer ID exists" or "whether it was processed within 24 hours." Conversely, attributes requiring human reading and judgment, such as "whether reasons align with criteria," still require sampling.

Sampling Methods

Samples should be selected to be representative of the population, and all items should have an opportunity to be selected (AS 2315 .24, .39). Random-based selection is a representative method. For tests of controls, it is desirable to use a selection method that allows items to be selected throughout the entire audit period (.39).

When subpopulations with different characteristics exist within a population, stratification is efficient. AS 2315 explains that dividing a population into relatively homogeneous groups based on characteristics related to the audit objective can reduce the required sample size, and an appropriate number of items is selected from each group (.22). When projecting the results of stratified sampling to the entire population, deviation rates are calculated for each group and combined according to the relative size proportions of the groups. Because sampling fractions differ across groups, one must not simply sum the total deviations across the sample and divide. Channels, transaction amount tiers, and departments with differing error histories are common criteria.

The Yellow Book establishes that selection methods depend on audit objectives. If a representative sample is needed, statistical sampling generally provides stronger evidence; if a representative sample is not needed, targeted selection based on risk factors may also be effective (GAO 8.107). Both statistical and nonstatistical sampling can provide sufficient evidence when properly applied (AS 2315 .03).

Factors Determining Sample Size

Sample size for tests of controls is determined by considering the tolerable rate of deviation (the maximum rate of deviation that can be accepted without altering planned reliance), the expected rate of deviation, and the risk of overreliance on controls (AS 2315 .38). For substantive tests of details, the auditor considers tolerable misstatement, the risk of incorrect acceptance, and the magnitude and frequency of expected misstatement (.23).

A few directions should be kept in mind:

• The higher the desired assurance—that is, the lower the acceptable risk—the larger the sample size. Relying solely on this test without other procedures requires setting risk low, resulting in a larger sample size (AS 2315 .19).
• The stricter the tolerable criteria, the larger the sample size.
• The number of items in the population has little effect on sample size unless the population is very small (AS 2315 Appendix Table 1 summarizing factors for substantive tests). If the number of processed cases quadruples, there is no need to quadruple the sample size.
• If the expected rate of deviation exceeds the tolerable rate of deviation, no increase in sample size can support the planned reliance. In such cases, planned reliance on controls should be revised and procedures directly responding to risks should be expanded (.43).

Evaluating Sample Results

The sample deviation rate is the best estimate of the population deviation rate. However, even if sample results are below the tolerable rate of deviation, the auditor must consider sampling risk—the risk that the actual population deviation rate exceeds the tolerable rate of deviation (AS 2315 .41). AS 2315 illustrates that if the tolerable rate is 5%, observing no deviations in a sample of 60 items may allow the conclusion that sampling risk is acceptably low, whereas observing two or more deviations may indicate that sampling risk is unacceptably high. The smaller the sample size, the greater the sampling risk (.10). If deviations fall between the two examples—such as observing 1 deviation—the auditor must separately judge the risk that the actual deviation rate exceeds the tolerable rate even if the sample deviation rate (1/60) is below the tolerable rate, and if necessary, increase the sample size or obtain additional evidence.

There are two common mistakes in evaluation. First, sample items that cannot be verified due to lack of evidence. If neither planned procedures nor appropriate alternative procedures can be applied, the auditor reviews the reasons and generally treats the item as a deviation (.40). Omitting cases with missing logs or replacing them with other items makes the results look better than they actually are. Second, the qualitative aspects of deviations. Along with frequency, the nature and cause of deviations must be evaluated—namely, whether it was a simple mistake, a misunderstanding of instructions, or an intentional circumvention (.42).

Designing Logs Backward from Audit Questions

Logs from AI systems are the most vital source of evidence for auditors. However, because logs are typically created by developers for troubleshooting, they often fail to answer audit questions. The GAO framework advises cataloging models and non-model components along with operating specifications and parameters (Practice 3.1), and documenting monitoring results and corrective actions to enhance traceability (Practice 4.3).

Retention periods are also part of log design. If company policy mandates keeping basis of determination records for several years but log retention is set to 30 days, that in itself constitutes a control deficiency and imposes a limitation on audit evidence.

Reproducibility and Reperformance

Whether reperformance is meaningful depends on the nature of the system. For a deterministic model that yields identical results for identical inputs, such as a tree-based scoring model, if the approved version is archived, having the auditor re-run the same inputs through the same version and compare the results provides the most direct evidence of whether "the recorded score originated from the approved model."

Generative models are typically different. For example, Anthropic's Messages API documentation notes that even when temperature is set to 0.0 on models supporting temperature adjustment, the results are not fully deterministic, and that newer models may not support this parameter at all. If running it again now produces a result different from the record, that does not prove the record is wrong; nor does producing an identical result prove it was right. Therefore, for decisions involving generative AI, reperformance is not about "whether rerunning it yields the same output," but rather "whether the judgment conformed to criteria based on the recorded inputs and outputs." To make this possible, raw input text, raw output text, and version information must remain in the logs.

Evidence Produced by the Audited Organization's AI

Audited organizations increasingly use generative AI to summarize meeting minutes, summarize customer service records, and draft reports. When such materials are submitted as evidence, two points must be kept in mind. First, summaries are materials derived from originals, and original documents possess higher reliability than transformed materials (AS 1105 .08). Second, generative AI carries the risk of confabulation (commonly referred to as hallucination), confidently producing plausible yet incorrect content (NIST AI 600-1). An AI meeting minutes statement claiming "unanimous committee agreement" only becomes evidence when corroborated against original recordings or electronic approval records. If evidence appears inconsistent, the auditor must perform procedures to resolve the inconsistency and determine the effect on other audit procedures (AS 1105 .29).

AI Tools Used by Auditors

Auditors also use AI. When searching for problem cases across tens of thousands of customer service records or extracting specific clauses from contracts, generative AI classification tools provide immense help. However, results generated by the auditor do not automatically possess reliability. Four steps should be observed:

1. Define decision criteria in writing first. Criteria are necessary to construct a validation sample and instruct the tool.
2. Measure errors by category using a validation sample. Compare the tool's results against a representative sample evaluated directly by the auditor to separately measure the false detection rate and the miss rate. In tasks searching for rare violations, many violations may be missed even if the overall misclassification rate is low; therefore, knowing the miss rate (the false negative rate in Chapter 4) is necessary to draw conclusions even about items classified as 'Not Applicable.' Also verify the tool's data processing terms before inputting audit target data.
3. Apply to the entire population and verify flagged items against raw source documents. Only classify the full population when both the miss rate and the false detection rate are within acceptable limits, and have the auditor verify items classified by the tool as 'Applicable' against raw source documents before using them in conclusions.
4. Document execution conditions in the workpapers. The Yellow Book requires documentation such that an experienced auditor having no previous connection to the audit can understand the nature, timing, extent, results and evidence, and conclusions of procedures from the workpapers alone (GAO 8.132). Document prompts, model versions, execution date and time, and the number of input items.

The fact that running the same prompt twice yields the same result, or a vendor's general benchmark score, does not prove accuracy for this specific task.

Going deeper

Why sampling is still necessary in an era of full-population analysis

As data analytics tools have improved, we often hear people say, "Sampling is no longer needed now; we can just examine everything." This is half true. For attributes that can be determined by rules, full-population analysis is more complete and efficient than sampling. Sampling risk is eliminated, and not a single exception is missed.

However, full-population analysis also has limitations. First, analysis only looks at what the data tells it. An analysis result showing that a reviewer ID exists for every case merely demonstrates the formality that a review was performed; it does not show whether the judgment was correct. The quality of judgment still requires sample reperformance where a human reads the source materials. Second, the conclusion of the analysis relies on the completeness and accuracy of the data. If the extraction conditions were incorrect or a certain period was omitted, the term "full population" gives false confidence instead. Third, if an AI classification tool was made to "read" the entire population, the risk arises that the tool may misjudge violations. Such a risk of failing to detect errors even after applying procedures to all items is nonsampling risk, not sampling risk (AS 2315 .11), and when estimating the misclassification rate with a validation sample, sampling risk applies to that estimation once again. Ultimately, a good design combines the two. Formal attributes are checked through full-population analysis, the quality of judgment through sampling, and the reliability of the tool through a validation sample. In any case, the completeness and accuracy of the source data must be verified before analysis.

Integrity of agent logs

In systems where an AI agent places orders and modifies records on its own, the question of log reliability becomes even sharper. This is because the entity producing the record is the very subject of the audit. If logs are stored on the server where the agent runs, and the agent or the operator has write permissions to that storage, the log is closer to "a diary written by the agent itself."

AS 1105 explains that information produced by the company is more reliable when controls over that information are effective (.08). When applied to agent logs, this leads to two questions: Are the logs recorded in separate storage that cannot be modified by the agent and the operator? And do the logs reconcile with independent external records, such as order records or payment records in a counterparty's system? Asking an agent to summarize its own activities merely obtains another statement from the same system; it is not independent confirmation.

At work

Many Pines Card: Same control, different samples

At the credit card company Many Pines Card, when an AI either automatically approved a request for a temporary credit limit increase or routed it as "Needs Review," a reviewer judged the "Needs Review" cases. The audit team sought to verify two things regarding the cases processed over the past year. One was whether, under the policy, all "Needs Review" cases were processed within four hours along with a reviewer ID; the other was whether the reasons recorded by reviewers when approving cases met policy criteria.

For the first question, one team member proposed a random sample of 60 items. Citing the fact that the attributes to be verified were structured values—processing time and ID—and that all 94,000 processed cases were in system tables, the team leader opted for full-population analysis. Before the analysis, however, the audit team reconciled the count of the data extracted directly by the audit team against monthly processing reports, discovered that two days of data in March were missing, and re-extracted it. The full-population analysis revealed 312 cases that exceeded four hours, and the audit team reclassified these cases by cause.

The second question required a human to read the reasons. The population consisted of 18,000 approved cases, among which 2,000 cases were overseas merchant transactions, an area where omitted reasons had been concentrated in the previous audit. With a simple random sample of 60 items, only about seven overseas cases would be selected on average. The audit team used stratified sampling, separating overseas and domestic transactions to draw samples from each. Some suggested selecting only overseas cases that showed risk signals, but the results obtained that way could not be projected to the entire population of approved cases, failing to meet the audit objective.

During the evaluation phase, one domestic sample item could not be verified because an attachment containing the rationale had disappeared during a system migration. There were no other records to corroborate the same details. The audit team did not exclude that item, evaluating it as a deviation instead, and recorded the loss of the attachment as a separate control deficiency. The workpapers documented the reasons for choosing the method for each of the two questions, the discarded alternatives, and the results of the data completeness check.

Spring World Corporation: How to reperform generative answers

At the public agency Spring World Corporation, generative AI drafted responses to inquiries regarding eligibility for rental housing, which counselors reviewed and sent out. The audit team sought to verify whether "responses providing incorrect guidance on eligibility criteria had passed through review."

One team member suggested feeding several inquiries from the previous quarter back into the current system to see if the same answers were produced. The audit team leader rejected this for two reasons: generative models can produce different outputs even for the same input, meaning that the rerun results cannot be used to judge whether past records were right or wrong; furthermore, the knowledge documents had been updated twice since the previous quarter, so the current system was no longer the system from that time.

The audit team checked the logs first. The logs retained the original inquiry text, the original AI draft, the final response sent by the counselor, the versions of the instructions and knowledge documents, and the counselor ID. However, because the retention setting was 90 days, some records from the beginning of the quarter were about to reach the end of their retention period and be deleted. The audit team first requested that the records be preserved, and then drew a sample from 41,000 inquiries. The auditor then re-judged the responses sent by the counselors against the versions of the knowledge documents and public announcements in effect at that time. This was reperformance of a control involving generative AI.

In 3 out of the 60 sample items, responses that provided incorrect guidance on income criteria had been sent out as is. In all three cases, the counselors had failed to correct errors in the AI drafts. The audit team evaluated this result along with sampling risk, and raised the log retention setting—which was shorter than the retention period for decision rationales (three years)—as a separate finding.

Common mistakes

1. "With tens of thousands of data records, the evidence is sufficient." Quantity is only one aspect of sufficiency. Low-quality evidence cannot be compensated for with quantity.
2. "Because the auditor extracted it directly, there is no need to check completeness." For computer-processed information, completeness and accuracy must be evaluated regardless of who extracted it (GAO 8.98).
3. "The overall error rate can be determined from the results of examining only high-risk items." The results of selecting specific items cannot be projected to the population (AS 1105 .27).
4. "If the population quadruples, the sample size must also quadruple." Unless the population is very small, its size has almost no effect on sample size.
5. "Sample items that cannot be verified should be excluded from the evaluation." If alternative procedures cannot be applied either, they are generally treated as deviations (AS 2315 .40).
6. "If the sample deviation rate is less than or equal to the tolerable deviation rate, it passes." Because of sampling risk, the possibility that the actual deviation rate exceeds the tolerable rate must be considered. Even if it appears low, such as 1 out of 60 items (1.7%), judgment is required.
7. "If rerunning generative AI yields the same answer, the record is correct." Generative models are not entirely deterministic. Judgment is reperformed using the recorded inputs and outputs.
8. "AI meeting minutes or AI summaries are more reliable because they contain no human error." Summaries are derived materials created from the original source and may contain plausible inaccuracies. Cross-check against the original source.

Summary at a glance

Check the official docs

• PCAOB — AS 1105 Audit Evidence: Sufficiency and appropriateness (.04–.08), information produced by the company (.10), procedures (.15–.21), selecting items (.22–.28), conflicting evidence (.29).
• PCAOB — AS 2315 Audit Sampling: Stratification (.22), representativeness (.24, .39), sample size (.19, .23, .38, Appendix Table 1), evaluation (.40–.43).
• GAO — Government Auditing Standards 2024: Evidence (8.90–8.115), documentation (8.132–8.135).
• GAO-21-519SP: Practice 3.1 (Component inventory), 4.3 (Traceability).
• NIST AI 600-1: Confabulation and automation bias.
• Claude Docs — Messages API: Description of the temperature parameter and notes on determinism.

Self-check

3-1. State the three elements of appropriateness according to the Yellow Book, and explain in which of those elements the plan to "judge the accuracy of AI answers using customer satisfaction scores" is weak.

3-2. State the two methods for using information produced by the company as evidence, and give one appropriate situation for each.

3-3. State the limitation that must be observed when reporting the results of examining only cases that exhibit risk signals.

3-4. State the three factors that determine the sample size for tests of controls, and explain in one sentence the effect of population size on sample size.

3-5. In a test of a sample of 60 items with a tolerable deviation rate of 5%, one deviation occurred. State two things the auditor should consider together before reaching a conclusion.

3-6. Explain why the methods for reperforming the results of a deterministic scoring model and a generative summarization model differ, and state three log items required for reperforming a generative model.

3-7. (Design) An audit team intends to classify 20,000 internal whistleblower reports using generative AI to identify suspected fraud cases. Design a four-step procedure for using these results as audit evidence.

Hands-on tasks