Chapter 4. Measuring AI Content Performance

Learning objectives

• Turn performance measurement questions into testable forms, and establish the definitions of key metrics (numerator, denominator, unit) and baselines.
• Design experiments that fairly compare AI content and human content through randomized assignment and concurrent comparison.
• Explain the principles of determining sample size based on significance level, statistical power, and the minimum detectable difference.
• Identify measurement errors such as repeated peeking, sample ratio mismatch, multiple comparisons, the novelty effect, and ignoring guardrail metrics.
• Select code-based grading, human grading, and LLM-based grading according to the nature of the criteria, and verify the reliability of LLM grading beforehand.
• Rewrite exaggerated interpretations in performance reports to stay within the measured scope.

Introduction

"Switching to AI copy increased click-through rate by 40%." A report like this immediately lifts the mood in the conference room. But ask a few questions, and the mood shifts. Did you compare last month to this month, or did you split and send during the same period? What was the click-through rate divided by? Were there any discount promotions in between? How many people was it sent to, and when did you check the results? What happened to opt-outs?

AI has made it possible to create infinite content, making the task of discerning "which one is truly better" all the more critical. However, clumsy measurement either exaggerates the effects of AI adoption or, conversely, misses the real effects. What is more dangerous is that flawed measurement produces flawed learning. If copy that won by chance is treated as 'The correct answer' to craft subsequent prompts, the entire organization ends up following random chance.

Measurement also intersects with the law. Messages sent to an experimental group are still advertisements; if opt-outs increase, there will be customers to whom messages cannot be sent again; and using customer data for quality evaluation constitutes personal information processing. Therefore, the measurement principles in this chapter operate on top of the rules of Chapter 1 and Chapter 3.

This chapter covers the basic grammar of experimentation (questions, metrics, baselines, randomized assignment), the bare minimum of statistics (significance level, statistical power, sample size), measurement errors that frequently arise in practice, and methods for evaluating the quality of AI content. It centers on judgment rather than formulas, and presents only principles that can be confirmed through official documentation.

Key concepts

What are you trying to prove

Measurement begins with a question. "Is AI copy good?" is not a testable question. Transforming it into a testable form looks like this: "In app pushes sent to the same customer segment during the same period, does the AI-written copy achieve a higher 7-day purchase conversion rate than the existing copy?"

A good experimental question has four elements.

Define a single key metric before the experiment. Selecting whichever looks good among multiple metrics after the experiment concludes is not measurement; it is post-hoc rationalization.

Metric definitions: Numerator, denominator, and unit

Even metrics with the same name cannot be compared if their definitions differ. Taking click-through rate as an example, the value can vary by several times depending on whether the denominator is the 'Number of sends' or 'Number of opens'. Open-based click-through rate only includes people who have already shown interest in the denominator, so it comes out higher than when based on sends.

The unit must also be aligned. Statsig documentation explains that the unit of randomization (user, device, etc.) is always shown the same variant for the same value, and that unit becomes the baseline unit of the metric. If assignment was made at the person level, metrics must also be aggregated and compared per person. At this time, 'Percentage of people who clicked' counts duplicate clicks by a single person once, while 'Clicks per person' counts all repeated clicks. Both can be used, but you must decide which one to use and apply the same definition to both groups.

Baselines and success criteria

A baseline is the starting point of comparison—that is, current performance. Claude evaluation documentation advises setting success criteria that are specific, measurable, and achievable, citing as an example a certain percentage improvement over the current baseline. Baselines are used for two purposes.

1. Sample size calculation: Depending on whether the baseline conversion rate is 2.0% or 10%, the sample size required to detect the same improvement differs.
2. Decision-making criteria: Determine in advance "how much better than the baseline it needs to be to make a switch worthwhile."

A pitfall to watch out for is using the baseline as the comparison target. A design such as "Last month's performance is our baseline, so let's compare it with this month's AI copy performance" causes everything that changed between the two months (seasonality, promotions, prices, competitive landscape) to be mistaken for the effect of the AI copy.

Randomized assignment and concurrent comparison

The essence of a fair comparison is ensuring that the two groups are identical in every respect except for the copy. That method is randomized assignment. Statsig documentation explains that users are divided into a control group and one or more treatment groups to receive different variants, with randomized assignment ensuring that each group is evenly represented to compare variant performance.

What randomized assignment protects against is unobserved differences. If AI copy is sent to VIP members and human copy to regular members, differences in purchasing power appear as the effect of the copy. If AI copy is sent in the evening and human copy in the morning, time-of-day effects become confounded. When recipients are randomly divided and messages are sent at the same time through the same channel, these differences are dispersed equally across both groups.

Changing only one thing at a time

Statsig documentation advises against introducing more than one change into a single experiment in an A/B Test. Confounded changes that frequently occur in AI copy experiments include the following:

• Changing images, coupon amounts, or send times along with the copy
• Incorporating personalization data only on the AI copy side (in this case, what is being compared is not 'AI vs. Human', but 'Personalized vs. Non-personalized')
• Sending AI copy through a new channel (e.g., app push) and human copy through an existing channel (email)

Decide first what you want to compare. If you want to see the effect of 'The AI-written copy itself', you must provide the same information and only vary the author; if you want to see the effect of 'The entire AI personalization campaign', you must define that entire bundle as a single treatment and report it as such.

Significance level, statistical power, and sample size

Statistical testing carries two risks of error. The NIST/SEMATECH e-Handbook of Statistical Methods explains it as follows. The risk of concluding there is a difference when in reality there is none is called α (significance level), and tests are designed to keep this risk small. The risk of failing to detect a difference when one actually exists is called β; this risk depends on factors such as the size of the true difference, and for a fixed sample size, reducing α increases β. 1−β is called statistical power. Increasing the sample size allows both risks to be reduced simultaneously.

Familiarize yourself with two tools used when reading results. The p-value is the probability of observing a difference as large as (or more extreme than) the current one assuming there is genuinely no difference, and if it is smaller than α, the result is judged to be Statistically significant. A confidence interval represents the plausible range within which the true difference lies; if the 95% confidence interval for the difference between the two groups does not include 0, it is considered Statistically significant at the 5% significance level.

The sample size is determined by these two risks and the minimum difference to detect (δ). In the example from the NIST handbook, when attempting to detect an increase in defect rate from 10% to 20% using a one-sided test with α = 5% and β = 10%, the minimum sample size is calculated as approximately 102, or 112 with continuity correction. This example is a one-sample test comparing the defect rate of a single process against a baseline value. An A/B experiment comparing two groups reflects the variance of both groups, so even under the same conditions, a larger sample size is required. This example provides three intuitions.

1. The smaller the difference you want to detect, the larger the required sample size. Because sample size is roughly inversely proportional to the square of the difference to detect (δ), cutting the difference in half requires nearly four times the sample size.
2. For a metric with a low baseline value (e.g., a 2% conversion rate), detecting a small improvement (0.2%p) may require tens of thousands of people per group.
3. In an experiment with an insufficient sample size, "Not statistically significant" means "Insufficient evidence to judge," not "No effect."

Repeated peeking and sequential testing

Once an experiment begins, it is tempting to check the results daily. The problem lies in making decisions based on those results. Statsig documentation explains that traditional A/B Tests (primarily z-tests for proportion metrics like conversion rates and t-tests for continuous metrics like revenue) are fixed-horizon tests designed to make a determination only once when reaching a predetermined sample size, and that continuous peeking inflates the false positive rate above the established significance level. In frequentist procedures, early decisions act solely to increase the false positive rate.

The solution is one of two options: defer drawing conclusions until reaching the precalculated sample size, or use sequential testing, which is designed from the outset with repeated peeking in mind. Sequential testing adjusts p-values and confidence intervals so that early results can be interpreted. Looking at interim results in itself is fine; checking whether data is corrupted is in fact necessary. The problem is declaring a winner based on interim results in a fixed-horizon test.

Sample ratio mismatch (SRM)

If you split 50:50 but the actual participant counts come out to 41:59, you must stop before reading the results. Statsig documentation defines Sample Ratio Mismatch (SRM) as a state where the participant count in each group differs from the configured ratio beyond what can be explained by chance, and explains that it is detected using a chi-squared test. SRM is usually not random, so the people who were dropped or added are not identical to the original group.

The causes cited in the documentation are as follows: bugs where the app or browser crashes before sending exposure logs, conditions that filter out exposures only in a specific group, and scripts that send exposure logs sequentially by group where logs are cut off once a certain threshold is exceeded. In an AI copy experiment, one can also imagine cases where exposure logs for a specific device or app version are omitted in only one variant. Do not use results to draw conclusions until the cause is found.

Multiple comparisons

Suppose you use AI to create twelve copy variants, compare each with the existing copy, and find that one turns out to be Statistically significant. The NIST handbook explains that repeating comparison procedures across all possible pairs changes the overall significance level compared to when testing a single pair. The more comparisons you make, the higher the probability that at least one 'Statistically significant' result emerges by chance.

There are three ways to respond. Reduce and establish the comparisons to be made before the experiment. Adjust the significance level considering the number of comparisons (the handbook introduces methods such as the Bonferroni method for planned contrasts; the Bonferroni method divides the overall significance level by the number of comparisons to make the threshold for each comparison more stringent). And verify the winning variant again with a fresh experiment. The more easily AI generates many variants, the larger this problem becomes.

The novelty effect and experiment duration

A new copy or a new banner may initially elicit a heightened response due to curiosity, which then diminishes over time. This is called the novelty effect. Statsig documentation recommends running experiments for a longer duration to account for the novelty effect and adaptation time.

The way to judge is to look at the trend, not the average. If the weekly relative increase is declining from 35%, 14%, 6%, down to 2%, a 4-week average of 14% may overestimate the long-term effect. Because weekly figures also have uncertainty, look at the trend together with confidence intervals. Extend the duration until the effect flattens, or make judgments based on that leveled-off rate.

Guardrail metrics

A guardrail metric is a metric monitored to ensure that business health is not harmed while driving up the key metric. The Statsig glossary describes guardrail metrics as metrics designed to measure and maintain business health.

Representative guardrail metrics in AI marketing include opt-out rates, app notification disablement rates, spam reports, return rates, increases in customer inquiries, and brand tone violation rates. Predetermine acceptable limits for each guardrail metric before the experiment. The opt-out rate in particular carries major legal significance. Under Article 50 (2) of the Act on Promotion of Information and Communications Network Utilization and Information Protection, advertisements cannot be sent again to individuals who have opted out; therefore, an increase in opt-outs represents a cost that shrinks advertising reach until fresh Explicit consent is obtained. If provocative AI copy increased click-through rates while also driving up opt-out rates, you must not make the switch based on click-through rates alone.

Statistical significance and business significance

When sample sizes are very large, even very small differences become statistically significant. If sending to 1,000,000 people yields a click-through rate that is 0.03%p higher and turns out to be Statistically significant, that difference merely means "it is likely not chance"; it does not mean "it is worth making the switch." The magnitude of the effect must be weighed alongside the generation and review costs of the AI copy and operational risks.

That is why establishing the minimum commercially meaningful difference before the experiment is important. This value becomes the δ in sample size calculations and serves as the criterion for judgment when reading results.

Evaluating AI content quality: Code, human, and LLM

Performance experiments measure customer responses, whereas quality evaluation measures whether the content itself adheres to standards. If AI generates thousands of copy pieces every week, quality evaluation must also match that scale. Claude evaluation documentation divides grading methods into three:

The documentation gives three tips for LLM-based grading. Provide a detailed, clear rubric. Receive output in a concrete format, such as 'Pass/Fail' or a score of 1–5. Have the grading model reason before producing a score. In addition, the documentation cites evaluating with a different model from the one that created the evaluated output as a general best practice. This is because when the same model grades its own copy, it is prone to repeating the same misconceptions.

Rewriting performance reports

Exaggeration in reports generally takes five forms.

The principle is "State conclusions only within the measured metrics, target population, and time period." Writing 'Not verified' for what has not been confirmed is honest reporting.

Going deeper

What Does the 'AI vs. Human' Comparison Compare?

In practice, an 'AI copy vs. Human copy' experiment is harder to define than one might think. If a human edited an AI draft, is that AI copy or human copy? If you change the prompt, even within the same 'AI copy', it becomes a different treatment. Human copywriters also do not produce the same quality every time.

Therefore, when designing an experiment, it is better to define the treatment as a working method. For example, write it as "copy written by an AI using brand guide v3 instructions and reviewed only for factual claims by the person in charge" vs. "copy written by a copywriter from scratch". The results should record not only the response to the copy, but also production time, review costs, and guardrail metrics to be usable for decision-making. Even if the response to the AI copy is the same, if the production cost is halved, that is also a meaningful conclusion.

Experimental Unit for Personalized Content

When an AI writes different copy for each customer, a comparison of 'Copy A vs. Copy B' does not hold. What is being compared here is not individual copy, but the way copy is generated (the policy). The treatment group consists of "people who receive copy generated by the personalization policy," the control group consists of "people who receive common copy," and the assignment unit is the individual person.

There are two points to watch out for in this design. First, because the copy varies for each person within the treatment group, this experiment cannot determine "which sentence was effective." Second, because the data used for personalization is part of the treatment, the legal ground for using that data (Chapter 3) must be verified for the entire treatment group.

Experiments Are Also Advertising and Personal Information Processing

Messages sent to an experimental group are also advertisements. The fact that it is an experiment does not waive the substantiation obligations under the Act on Fair Labeling and Advertising or the transmission rules under the Act on Promotion of Information and Communications Network Utilization and Information Protection. If you want to 'Test' more aggressive copy, any factual claims contained in that copy must already be capable of substantiation, statutory exceptions such as opt-in consent or existing business relationships must be confirmed for each experiment subject, and media-specific nighttime rules must be observed. If customer information is transferred to an external tool for experiment data analysis, distinguish whether it constitutes outsourcing of processing or provision to a third party, apply the corresponding requirements, and verify whether a cross-border transfer takes place. Measurement rigor and legal compliance must be documented together in the same plan.

At work

One-Bite Bagel: App Push Copy Experiment

The app marketer at the bagel franchise 'One-Bite Bagel' wanted to test morning push copy written by an AI. The initial proposal was: "Starting next Monday, let's send AI copy to all members and compare it with last week's performance." The data specialist rejected the period-over-period comparison, pointing out that a new menu launch overlapped with next week.

The two co-authored an experiment plan. The target audience was members who had opened the app within the last 60 days, assignment was 50:50 randomized at the member level, and delivery was scheduled for the same day and time (8:30 AM, after the 9:00 PM to 8:00 AM nighttime window ended). Only a single key metric was selected: the mobile order conversion rate (based on delivery) up to 11:00 AM on the day of delivery, while the opt-out rate and the app notification turn-off rate were set as guardrail metrics. The baseline conversion rate was 3.2%, and the team considered an increase of 0.4%p or more to be worth adopting. Matching the required sample size calculated from these values, the experiment duration was set to two weeks.

On the third day of the experiment, when the conversion rate on the AI copy side reached 4.1% and took the lead, the team lead wanted an early termination. The data specialist explained that stopping a fixed-sample design mid-stream increases the risk of false positives, and instead checked only whether the assignment ratio matched 50:50. The ratio was normal. Two weeks later, the result showed a 0.5%p increase in conversion rate (the confidence interval did not include zero), and no difference in opt-out rates. The report recorded both the scope—"target of active members in the last 60 days, two-week experiment"—and the weekly trend (the effect was sustained over the two weeks). A proposal to test eight AI-generated variants simultaneously was postponed to the next experiment due to multiple comparison issues.

Travel Connect: Introducing LLM Scoring for Newsletter Copy

The travel platform 'Travel Connect' generated 2,000 newsletter subject lines and introductory copy each week using AI, and two brand managers had been reading only a sample of 200 of them. The managers wanted to use an LLM to score the rest as well.

First, the criteria were divided. 'Whether prohibited expressions (lowest price guarantee, 100% refund, etc.) are included' and 'whether the subject line is within 40 characters' were decided to be checked via code. 'Whether it fits the brand tone' and 'whether there are exaggerated promises' required judgment, so they were designated for LLM scoring. The manager wrote out the brand tone as a rubric so it could be divided into 'Pass/Fail'. These included items such as "Do not speak informally to readers," "No more than one exclamation mark per sentence," and "Do not use unverified figures."

Next, LLM scoring was run with the same rubric on 400 pieces of copy that had already been scored by two managers. The agreement rate was 81% on the first attempt. Reviewing the cases of disagreement revealed that the 'Exaggeration' item in the rubric was ambiguous. After revising the item with examples, the agreement rate rose to 93%, exceeding the standard (90%) pre-determined by the team. In datasets where most decisions are Pass, simply predicting 'Pass' for everything can result in a high agreement rate, so the manager separately verified the rate at which the LLM also caught copy as Fail when humans judged it as 'Fail' (87%). Only then was LLM scoring applied to all 2,000 pieces, using a model different from the one that generated the copy for scoring. A rule was also instituted for managers to re-score a random sample of 100 items every month to check that the agreement rate did not decline. The proposals to apply it to everything from the start and to have the model that generated the copy score its own output were rejected due to the risks of premature expansion before verifying reliability and self-evaluation, respectively.

Common mistakes

1. "Comparing last month and this month reveals the effect of AI." Everything that changed between the periods becomes confounded. You must randomize across the same period and compare concurrently.
2. "Click-through rate is click-through rate." If the denominators (sent vs. opened) and units (people vs. clicks) differ, they cannot be compared.
3. "If it is significant mid-way through, you can end early." Reading and deciding on a fixed-sample test mid-way increases false positives. Use sequential testing or wait until the planned sample size.
4. "If it is not significant, there is no effect." If the sample size is insufficient, you cannot detect an effect even if there is one. Recalculate the sample size using the difference you want to detect.
5. "Even if the assignment ratio is slightly off, the results can be used." A sample ratio mismatch is a signal of unknown bias. Do not use the results for conclusions before identifying the cause.
6. "The more variants you create, the better copy you will find." The more comparisons there are, the more accidental winners emerge. You must apply corrections or re-verify.
7. "If it is statistically significant, it is worth adopting." In large samples, even tiny differences are significant. Look at the effect size and cost together.
8. "LLM scoring is as accurate as a human, so it can be applied to everything right away." Provide a rubric, verify agreement with human scoring first before scaling up, and do not have the model that generated the copy score its own output.

Summary at a glance

Check the official docs

• NIST/SEMATECH e-Handbook — 7.1.3 What are statistical tests?: The meaning of and relationship between α and β.
• NIST/SEMATECH e-Handbook — 7.2.4.2 Sample sizes required: Examples of calculating sample size using target change δ, α, and β.
• NIST/SEMATECH e-Handbook — 7.4.7 How can we make multiple comparisons?: The problem of repeated comparisons altering the overall significance level and correction methods.
• Statsig Docs — Experiments Overview: Control and treatment groups, random assignment, assignment unit, one change at a time.
• Statsig Docs — Sequential testing: Fixed-sample tests, the peeking problem, and sequential testing.
• Statsig Docs — Sample ratio mismatch: Definition, detection, and causes of sample ratio mismatch.
• Statsig Docs (legacy) — Experiments Overview: Novelty effect and experiment duration.
• Claude Docs — Define success criteria and build evaluations: Success criteria, pros and cons of code, human, and LLM scoring, and tips for LLM scoring.

Self-check

4-1. Rewrite the question "Is AI copy better?" into a testable question, and indicate the four elements included in that question.

4-2. Explain why a report mixing open-based click-through rates and send-based click-through rates cannot be used for comparison.

4-3. State the three factors that determine sample size, and explain how the required sample size changes if the difference you want to detect is cut in half.

4-4. Give one example of what is permissible and one of what is not permissible when looking at results during an experiment, and explain the reasons.

4-5. The click-through rate of AI copy increased, but the opt-out rate doubled. Explain how to evaluate this result, referring to the concept of guardrail metrics and its relationship to the Information and Communications Network Act.

4-6. State three things to do before scoring brand tone with an LLM, in accordance with the recommendations in the Claude evaluation documentation.

4-7. (Design) Fill in the sections of an experiment plan comparing AI-personalized text messages with generic text messages (question, target audience, assignment, metrics, guardrail metrics, sample, duration, decision criteria, legal checks).

Hands-on tasks