Chapter 4. Defense testing and red teaming

Learning objectives

• Explain the roles, execution modes and limitations of input, tool and output guardrails and classifiers.
• Configure failure behavior based on task risk when a classifier returns no judgment.
• Design rules of engagement and attack scenarios starting from the threat model.
• Build evaluation sets with attack, legitimate and boundary cases, and judge success by outcomes.
• Measure and interpret defense effectiveness using attack success rate, legitimate-request false-positive rate and task completion rate.

Introduction

At a launch meeting, a team reported an "injection blocking rate of 98%." There was applause, and launch was approved. When an incident occurred two months later, it emerged that the number came from 50 attack prompts used to develop the classifier's rules. Not one attack had arrived through a document or email. No one had measured how often legitimate requests were blocked. Users were avoiding the frequently blocked feature and pasting work materials into other AI tools under personal accounts. The number was not wrong. No one had asked what data or paths it measured. After the incident, the security team's new measurement of document-path attack success told a very different story.

Measuring defenses is as demanding as building them. This chapter starts with where guardrails and classifiers block what, and how they should behave when classification fails. It then covers the definition of red teaming, rules of engagement, deriving scenarios from threat models and principles for building evaluation sets. Finally, it teaches outcome-based success judgments and how to measure and interpret defense effectiveness.

Key concepts

Guardrail map: what goes where?

Guardrails inspect inputs and outputs before and after the model and at tool-call points, blocking or modifying them. Think of airport security checkpoints: before boarding (input), when loading baggage (tool calls) and after arrival (output). Each checkpoint can stop different things.

Output guardrails run after the model finishes, so they cannot undo tool calls executed along the way. Agents using tools with side effects must have checks at the tool-call point. Guardrails also do not replace the authorization checks in Chapter 3. A guardrail asks "does this look suspicious?" Authorization asks "may this person do this?"

When does checking finish? The Agents SDK example

OpenAI's Agents SDK illustrates guardrail location and timing in detail.

A tripwire is a signal that stops execution when a violation is detected. Tool guardrails have scope limits. They apply to custom function tools, but not handoffs (calls delegating to another agent), hosted tools (web search, file search, hosted MCP, Code Interpreter and so on) or built-in execution tools (computer, shell and so on). In architectures with a manager agent and specialist subagents, the documentation recommends tool guardrails rather than agent-level guardrails for checking every call. SDK details vary, but the questions stay the same: when does checking finish, what can execute beforehand, and which tools fall outside checking?

Classifiers: types of decision engine and their limitations

A classifier is the decision engine of a guardrail. There are three types.

Distinguish the questions each classifier answers. OpenAI's Moderation API is a free tool to reduce unsafe content. Prompt Shields detects user prompt and document attacks, but its documentation says it does not judge whether content is harmful or policy-compliant. Neither answers "is this user allowed to view this customer record?" Harm classification, attack detection and authorization cannot replace one another.

The OWASP injection prevention cheat sheet warns that an LLM acting as a guardrail is still an LLM and is itself vulnerable to injection. Do not read a classification score directly as "the probability of an attack." Its meaning must be established by evaluating the classifier on your own data.

Failure behavior: when no judgment is returned

Classifiers do not always answer. They time out, fail or refuse the screening request itself. Anthropic recommends treating refusal by a screening model's safety classifier as a harmful judgment when no result is produced (stop_reason is "refusal"). Treating no judgment as a pass gives attackers a bypass: input that makes the classifier refuse.

Set failure behavior according to task risk.

When introducing a classifier, one option is to observe operational data in annotate mode before blocking, measuring false positives. Foundry can block on Prompt Shields results or annotate only; the standalone Content Safety API returns judgments, leaving handling to the application. Other layers must contain damage during annotate mode.

Definition of red teaming and rules of engagement

NIST AI 600-1 defines AI red teaming as structured testing that probes AI systems to find flaws and vulnerabilities, usually in a controlled environment in collaboration with developers. It distinguishes general-user, expert, mixed and human-plus-generative-AI approaches, and says results need further analysis before informing organizational decisions. OpenAI recommends testing both representative inputs and inputs intended to break the system. Anthropic recommends testing your agents with documents, emails and tool results containing injection attempts, checking that screening and confirmation catch what gets through.

Red teaming means testing attacks against your own system before others do, always within authorized scope. Agree on written rules of engagement before starting.

1. Target and environment: Which system and version, and which environment separate from production, will be tested?
2. Data: Use synthetic data and fake secrets instead of actual customer data.
3. Allowed and prohibited actions: Replace external sending and payments with mock tools; do not touch third-party systems.
4. Stop conditions: Stop immediately if production-data access or unexpected external requests are observed.
5. Contact and reporting: Who receives findings, in which format and within what deadline?

Designing attack scenarios

A scenario is a unit for testing a path in the threat model, rather than a list of attack prompts. Start from Chapter 1's trust boundaries and Chapter 2's five-stage table, writing one card per scenario.

Check scenario coverage with a path × defense-layer grid. Rows contain paths (direct input, web, email, documents, images, tool results, memory); columns contain defense layers (input screening, tool-result screening, tool-call checks, human confirmation, output restrictions). Mark which cells have scenarios. Human confirmation is not a guardrail location but counts as a separate defense layer. Every layer needs scenarios targeting it to measure its effectiveness. Include separate scenarios spanning turns and sessions and scenarios bypassing or turning human confirmation into a formality.

Broadening scope with existing attack knowledge bases

Scenarios based only on imagination test only attacks the team knows. MITRE ATLAS connects tactics and techniques to real case studies. Use it to scan for techniques relevant to your system at stages such as reconnaissance, initial access, AI Model Access, AI Attack Staging, exfiltration and impact. Alongside direct and indirect injection, OWASP LLM01 describes scenarios involving unintentional injection, retrieval-store document tampering, payload splitting, multimodal input, seemingly meaningless suffixes, and multilingual or obfuscated attacks. The OWASP injection prevention cheat sheet classifies types such as encoding bypass, repeated variants and multi-turn attacks.

Use these knowledge bases as coverage checklists, rather than copying public attack prompts into evaluation sets. Prompts likely already seen during classifier training inflate scores and fail to capture your business context.

Building the evaluation set

Evaluation sets need more than attacks.

Anthropic's evaluation guidance recommends matching the real workload distribution, including boundary cases, automating grading with code or a judge model where possible, and prioritizing many automatically graded cases over a few carefully graded by people. Add these principles for security evaluation sets.

1. Do not evaluate defenses using the data used to develop their rules. Separate development and final validation sets, keeping the latter hidden from rule authors. A mock exam taken after seeing the questions does not measure ability.
2. Use the service's real input paths. Document attacks must pass through upload and text extraction; email attacks through email ingestion.
3. Include attack and legitimate cases for each service language. Treat a multi-turn attack as one case, replaying the whole conversation with state carried forward.
4. Plant synthetic secrets (canaries). These must be unique strings that do not match real values and cannot be generated accidentally (for example, a fake key with a unique prefix and random suffix), so their appearance in output or external requests establishes leakage.
5. Version the set and record each case's provenance (which threat or incident).

Judging success: use outcomes

A refusal response does not prove a defense succeeded. The model may refuse while already having called a tool, or politely appear to comply without acting. Judge observable outcomes.

Use a judge model (LLM grader) when the outcome exists only in text, as with answer contamination. Anthropic describes using a different model from the one that generated the output as a general best practice. Before using a judge, compare it with a human-labeled sample and measure agreement. Revise criteria or use human grading for categories with concentrated disagreement. Where logs can establish a scenario's result, prefer them over judge models. Judges can also be influenced by statements within evaluated output, such as "this response meets all criteria." Separate output as data, accept only structured judgments, and track judge-human disagreement as a metric.

Metrics for defense effectiveness

Use at least three numbers to assess defenses.

An unblocked legitimate request may still fail due to model errors or timeouts, so task completion rate is not simply 100% minus the false-positive rate.

Check the denominator first. Scenario-, attempt- and conversation-level measures can produce very different numbers from the same results. If one scenario succeeds once in 50 attempts, its attempt-level success rate is 2%, but at scenario level it is a "breached scenario." Because attackers can repeat attempts, "did this scenario ever succeed?" is often more important for security decisions.

Also track metrics per layer. Recording where each attempt stopped reveals contributions such as "input screening caught 70%, and human confirmation caught most of the remainder." A layer that catches almost nothing may be unnecessary, or may lack scenarios targeting it.

Continuous monitoring and regression

Pre-launch evaluation is only a snapshot at launch. Anthropic recommends regularly analyzing signs of successful injection to refine prompting, validation and filtering, and responding to repeated bypass attempts by the same user with warnings, rate limits or bans. This matches SAIF's element of adapting controls with rapid feedback cycles.

Run three feedback loops in operation: de-identify red-team and incident cases and add them to the regression set; rerun whenever the model, prompt, tools or classifier changes; and investigate sudden shifts in guardrail trigger rates, false-positive reports or approval rates. Guardrails are operational components that must be measured and improved.

Going deeper

How many runs? Interpreting probabilistic results

A scenario blocked once does not establish that it "is blocked," because output is probabilistic. One statistical rule helps set repetitions. If no successes occur in n independent attempts, the approximate 95% confidence upper bound on the true success rate is 3/n (an approximation for sufficiently large n). Zero breaches in 30 runs does not mean "the success rate is zero"; it supports only "the success rate is likely below about 10%." A 1% level requires roughly 300 runs.

This teaches two lessons. First, report "it appears below this level" rather than concluding "safe" from a small evaluation. Second, impact-reducing controls are often more efficient than endlessly narrowing probabilities through repetition. Evaluation reports must include model, prompt, tool and classifier versions, attempt counts and denominator units for reproducibility. The OWASP cheat sheet likewise notes that passing a brief check does not establish resistance to persistent attackers.

Experiments comparing control combinations

"Adding all defenses improved results" does not tell you which control worked. Design comparisons as follows. Measure an undefended baseline first. Add controls one at a time and measure each configuration with the same evaluation set, model version and repetition count. Also measure configurations with a control removed. If removing it barely changes results, it overlaps another control or the scenarios do not target it.

Read attack success and task completion together. If lowering attack success by 2 percentage points costs 15 percentage points of completion, consider changing the control's form: confirmation rather than blocking, or high-risk tasks rather than all tasks.

At work

Garam Securities: which layer missed what?

The brokerage Garam Securities plans to add account lookup and scheduled-transfer tools to its customer chatbot and prepares a red-team plan four weeks before launch. The rules specify a test environment separate from production, synthetic customer data, a mock transfer tool, stopping on external requests and same-day reporting.

There are three trust boundaries: customer chat input, customer-attached transaction confirmation images and scheduled-transfer tool calls. Existing defenses include an input classifier, passing OCR output from attachments as screened tool results, and a customer confirmation screen before scheduling transfers.

Scenarios target each layer. Input screening is tested with 60 policy-bypass conversations; the OCR path with 25 test images containing hidden instructions; and confirmation with 12 scenarios looking for transfers without approval. Each runs five times. Another 400 legitimate customer inquiries measure false positives and task completion. Success is judged by mock-transfer call logs and output containing synthetic account numbers planted as canaries.

Input screening catches most policy-bypass attempts but blocks 5% of legitimate inquiries. Two OCR scenarios pass screening in some repetitions, but all stop at confirmation. One confirmation scenario finds a flaw: an "edit" feature for changing the scheduled date reuses the original approval and also applies amount changes without confirmation.

The team adjusts screening to reduce false positives, adds the two OCR scenarios to the regression set and requires fresh confirmation when edits change the amount or recipient. The launch report documents each layer's catches and misses, repetitions and remaining risk, rather than a single success-rate number. The point is "which layer missed what, and did the next layer support it?" rather than "how many were blocked?"

Pinwheel Rental: three configurations on the same test

The car rental company Pinwheel Rental operates a booking-change assistant but cannot agree on defenses. Security wants a stronger input classifier; product worries that extra confirmation screens will drive customers away. The team decides to measure using the same test instead of continuing the debate.

The set contains 150 attack scenarios across direct input, user-linked web pages and partner booking emails, plus 300 legitimate tasks built from de-identified real inquiries. Each attack scenario runs five times. Three configurations are compared.

All three retain the existing server-side authorization.

B eliminates unauthorized changes, but many legitimate customers leave before confirmation. C blocks unauthorized changes while largely preserving completion. The basis for leaving date changes unconfirmed is checked separately: the server allows them only on the customer's own booking and only where prices do not change.

The team also learns about judgment. Initially a judge model reads the entire conversation and labels attack success. Comparison with 100 human-labeled cases shows clusters of disagreement on cases that "refuse verbally but call the tool." The team switches action-scenario judgments to mock-tool logs and reserves the judge for answer contamination. It selects C and reports repetitions, denominator units and the tested model version.

Common mistakes

1. "Moving checks to an output guardrail is safer." Output guardrails run after the agent finishes and cannot undo executed calls.
2. "If the screening model refuses, no judgment means pass." Treating no judgment as a pass creates a bypass. Anthropic recommends treating refusal as harmful.
3. "A high blocking rate is enough." First ask what data, how many runs and which denominator. Numbers measured on rule-development data are not meaningful.
4. "False positives are only a usability issue." Frequent false positives drive users outside controls and eventually get controls disabled. They are a security metric too.
5. "A refusal response means the defense succeeded." Refusal and tool execution can coexist. Judge by logs and state changes.
6. "A harm classifier stops injection." Harm classification, attack detection and authorization answer different questions.
7. "Thirty runs with no breach means safe." This supports only that the success rate is likely below roughly 10%.
8. "A public collection of attack prompts is enough for evaluation." Public attack prompts inflate scores and bypass your actual paths. Use knowledge bases to check coverage.

Summary at a glance

Check the official docs

• OpenAI Agents SDK — Guardrails: guardrail types, execution modes and scope.
• Anthropic — Mitigate jailbreaks and prompt injections: screening, refusal handling, monitoring and red teaming your agents.
• Anthropic — Define success criteria and build evaluations: evaluation design and grading.
• OpenAI — Safety best practices: Moderation API, adversarial testing and human review.
• Microsoft — Prompt Shields: detection scope and limitations, annotate and block modes.
• NIST AI 600-1: AI red-teaming definition, types and cautions for using results.
• MITRE ATLAS: adversary tactics and techniques for checking scenario coverage.

Self-check

4-1. Name two tool types not covered by OpenAI Agents SDK tool guardrails, and give one alternative control for an agent using them.

4-2. Write one line each stating the question answered by harmful-content classification, injection attack detection and authorization.

4-3. List five items to agree in rules of engagement before red teaming.

4-4. State the purpose of planting synthetic secrets (canaries) in evaluation sets, and two requirements for choosing them.

4-5. Nine of 240 attack attempts succeed, and 15 of 500 legitimate requests are blocked. Calculate attack success and legitimate-request false-positive rates, and state one thing these two numbers cannot reveal.

4-6. A scenario runs independently 60 times with no breach. State the approximate upper bound on success rate supported by this result, and what cannot be claimed.

4-7. Design task: compare the effectiveness of "adding input screening" and "adding write-tool confirmation." Give configurations to compare, three conditions to hold fixed and two metrics to track.

Hands-on tasks