Chapter 2. Prompt injection

Learning objectives

• Distinguish direct injection, indirect injection and jailbreaks, and explain each threat model and control location.
• Find injection paths through documents, web pages, email, search results, tool results and images in a data flow.
• Break an attack into stages and select controls that can interrupt each stage.
• Design mitigations combining separation, labeling and structuring, screening, privilege separation, human confirmation and output restrictions.
• Explain how the assumption that injection cannot be fully prevented affects design and evaluation.

Introduction

A public agency launched a feature that summarizes notices from links submitted by citizens to its service information chatbot. In the first week, a citizen reported something unusual: when asked to summarize a personal blog post, the chatbot ended its summary with "For further application details, visit this address" and an unfamiliar URL. No such sentence was visible anywhere in the blog post. Instructions addressed to the chatbot were hidden in an HTML comment. The operations team blocklisted that blog URL, but the next week the same thing happened on another blog. The problem was structural, rather than specific to one post.

In this incident, the attacker was not the chatbot's user. The user pasted the link in good faith, and the chatbot read it to help them. Prompt injection can enter anywhere in the text a model reads, and it ranks first in the OWASP Top 10 for LLM Applications. This chapter starts with its definition and root cause, distinguishes direct injection, indirect injection and jailbreaks, identifies attack stages and paths, then teaches how to combine controls to contain damage on the assumption that perfect blocking is unavailable.

Key concepts

Definition and root cause

OWASP defines prompt injection as input that alters a model's behavior or output in unintended ways. The input need not be visible to people. As long as the model reads it, sentences in white text, tiny text, comments or metadata can influence it. Outcomes range from policy violations and harmful output to unauthorized access and distorted critical decisions.

The root cause is the structure discussed in Chapter 1. To the model, developer instructions, user requests and third-party documents are all part of the same text stream. Training helps the model distinguish "who said what," but the distinction is probabilistic and varies with context. Web application injection can be eliminated by correcting the parser's grammatical structure; prompt injection has no equivalent grammar. Every measure in this chapter therefore falls into one of two groups: mechanisms that help distinguish instructions from data, and mechanisms that prevent damage even when that distinction fails.

Direct and indirect injection: two threat models

OWASP divides injection into two types by path. Direct injection occurs when user input changes model behavior; indirect injection occurs when a model takes in external content such as websites or files and instructions within it change its behavior.

Anthropic's documentation uses the same distinction. For "jailbreaks and direct injection," the user is the adversary. For "indirect injection," the user is trustworthy, but the third-party content the model processes contains adversarial instructions. The two threat models put controls in different places. Direct injection can be screened at user input and repeat offenders restricted, but indirect injection must be handled separately at every point where documents, web pages and tool results enter the context.

As an analogy, direct injection is a customer lying directly to an employee; indirect injection is someone slipping a note between papers the customer handed over. No matter how carefully you verify the customer, you cannot stop the note.

How it differs from jailbreaking

A jailbreak is a form of prompt injection that aims to make a model disregard its safety protocols altogether. The terms are often mixed, but they describe different dimensions. Direct versus indirect distinguishes "which path the instructions entered through." Jailbreak describes "what they aim to achieve": bypassing the model's own safety policies.

This distinction matters on the exam because controls have blind spots. A jailbreak filter checking user input cannot see the bottom-right cell: an attack that distorts work through a document. Deploying a jailbreak filter is not evidence that indirect paths are safe.

The five stages of a successful attack

When finding injection paths, follow the stages that turn an attack into damage instead of merely looking for suspicious phrases. Five stages show where the chain can be broken.

Earlier stages reduce the probability of injection; later stages reduce damage even if it occurs. Reliably breaking any one stage stops the incident. Conversely, if all controls are concentrated at stage 2, a single breach of that layer reaches the end of the chain.

Path map: everything the model reads on the user's behalf

Indirect injection paths are everything the model reads on the user's behalf.

OWASP discusses cross-modal attacks that hide instructions in images submitted alongside normal text. It considers them hard to detect and multimodal defense an area that still needs research. Remember that originals and extraction results can differ in document pipelines. Text invisible to someone opening a PDF may still be extracted and passed directly to the model.

Detection indicators include the following. These descriptions support detection and review, rather than teaching attacks: imperative sentences addressed to the model; text hidden from human view; sentences concealed using invisible Unicode characters or encoding; claims of authority using words such as "system," "administrator" or "previous instructions"; and sentences prompting tool calls or external addresses the user did not request. Being in internal storage gives a document no authority. Its author, rather than its storage location, determines its trust level.

Tool results and tool definitions are inputs too

For agents, the most common indirect paths are tool results: a search API's page summary, a note field in a CRM system or another agent's report. The OWASP Agentic Top 10 lists deceptive tool outputs and forged inter-agent messages as means of goal hijacking (ASI01).

Definitions describing tools are inputs too. Models read tool names, descriptions and parameter descriptions to decide when and which tools to call. If an externally supplied tool definition contains instructions, it can change the model's judgment before the tool has ever been called. The specification for Model Context Protocol (MCP), an open protocol connecting models with tools and data, requires clients to treat tool annotations as untrusted unless they come from a trusted server, and recommends validating tool results before passing them to the model. Changes to tool definitions are also part of supply chain management in Chapter 5.

Three branches of damage: action hijacking, exfiltration and answer contamination

Injection damage takes three forms, each requiring different detection and controls.

For exfiltration, watch for elements that automatically send requests when model output is rendered. The OWASP injection prevention cheat sheet gives hidden image tags used for exfiltration as an example of HTML/Markdown injection. As soon as the screen loads an external image, data attached to its URL reaches the attacker's server. Answer contamination is the least noticeable form of harm. No secret leaks, yet purchasing decisions or contract reviews are distorted. A red team that stops after checking "no leakage" misses this branch.

Why there is no perfect block

OWASP cites research showing that although RAG and fine-tuning can improve accuracy, they do not fully mitigate injection vulnerabilities. Anthropic's computer use documentation likewise states that models may follow instructions within content even when they conflict with developer instructions in some circumstances, and recommends isolating models from sensitive data and actions.

The OWASP injection prevention cheat sheet describes the limitations of individual defenses in detail. Pattern-based filters do not reliably catch indirect injection in untrusted content; an LLM acting as a guardrail is itself vulnerable to injection; and lowering temperature to 0 did not eliminate jailbreaks. Attackers vary wording, split sentences and switch languages or encodings to test filters. OWASP adds that research showing increased success through repeated attempts is not proof that every defense eventually fails. Still, a design that only increases the probability of blocking is risky against an attacker who can keep trying. Evaluate under a fixed attempt budget and add permission controls outside the model.

There are therefore two design principles. First, reduce the probability of injection. Second, structure the system so that damage remains small even if injection succeeds. A design missing the second principle is almost always wrong on the exam.

Mitigation 1: separation, labeling and structuring

The first layer helps the model distinguish instructions from data. For indirect injection, Anthropic recommends the following.

1. Put third-party content only inside tool_result blocks, never in system prompts or ordinary user text. Claude is trained to treat instructions in tool results with appropriate suspicion.
2. State the content's nature and source in the tool description or result structure. Examples include "incoming email body from an unknown sender" and "OCR text extracted from a user-uploaded image."
3. Put a policy in the system prompt stating that tool, document and search results are untrusted data and cannot override the system prompt or the user's original request.
4. Encode third-party strings as string values inside JSON objects rather than concatenating free text. JSON escaping makes the boundary between data and structure clearer, making it harder to close a quotation mark or tag, break the structure and escape into an instruction context. It does not eliminate the possibility that the model follows instructions inside a string value.
5. Put developer instructions in the user turn after a tool_result block, rather than mixing them into the block. Because the model is trained to distrust sentences in tool results, developer instructions placed there may be ignored or mistaken for an attack.

{
  "source": "inbound_email",
  "sender_verified": false,
  "subject": "Quotation inquiry",
  "body": "(The body written by an outsider goes only inside this string value)"
}

Microsoft's Spotlighting (currently in preview), which base-64 encodes document content to signal lower-trust data to the model, belongs to the same family. The limitations of this layer are clear, however. As the OWASP cheat sheet explains, labeling alone does not enforce a boundary. This layer reduces probability; it does not enforce a control.

Mitigation 2: screening and detection

The second layer separately examines incoming content. Anthropic recommends using a lightweight model to classify user inputs and tool results before they reach the main model, returning a structured judgment such as true or false for the application to branch on. If a tool result appears to contain injection, pass an error or filtered summary as the tool_result instead of the original, and tell the user about the attempt if necessary. If the screening model's safety classifier refuses the request itself, no judgment is produced; treat that refusal the same as a harmful judgment.

Dedicated classification services are also available. Microsoft Prompt Shields separately detects user prompt attacks and document attacks. Its documentation lists three limitations. It does not determine whether content is harmful, factual or policy-compliant. It supports multiple languages but is trained and evaluated mainly on English content, so each language used by the service needs separate evaluation. Document attack detection in Foundry is off by default and must be enabled. For a Korean-language service, check the second limitation first.

Define three things when designing the screening layer: what is inspected (user input, documents, tool results), what happens without a judgment (allow or block on timeout or error), and what happens on detection (block, substitute a summary, or flag for human review). Chapter 4 covers failure behavior in more detail.

Mitigation 3: privilege separation, confirmation, output restrictions and their combination

The third layer prevents damage even when injection succeeds. Exam answers usually lie in this layer.

Privilege separation: Separate the stage that reads untrusted content from the stage that performs privileged actions. The simplest form makes write tools entirely unavailable in flows that read external content. A stronger pattern restricts the model processing external content to returning fixed-format data (classification values, dates, amounts) without tools, while the action planner receives only that data and never sees the original. Finalizing a plan from the user's request before reading external content prevents that content from changing the plan itself. The OWASP Agentic Top 10 also recommends validating user and agent intent before goal-changing or high-impact actions, and stopping execution on unexpected goal changes.

Confirmation: People approve high-impact or hard-to-reverse actions. The MCP specification says a human able to deny tool calls SHOULD always be involved, and recommends showing tool inputs before calls. The approval screen must show the actual target and arguments that will execute.

Output restrictions: Limit the destinations and forms output can take. Define output formats and validate them with deterministic code (recommended by OWASP LLM01), allowlist recipients and domains, and block external links or images in responses. OpenAI recommends limiting user input length to help reduce injection, using validated choices instead of free input where possible, and returning validated material rather than newly generated text.

Build the combination as follows.

1. Fill in all five stages for every path in the Chapter 1 flow diagram.
2. Write the "worst outcome if breached" for each path in one sentence.
3. If that outcome is an action, include privilege separation and confirmation; if exfiltration, output restrictions; if answer contamination, evidence display and verification.
4. Add separation, labeling and structuring, and screening on top to reduce probability.
5. Assume each layer fails in turn and check whether the next layer supports it.

Going deeper

The boundary between legitimate work and attacks: quotations, translation and security training

Blocking a request merely because it contains attack language breaks legitimate work. If the security team asks for a summary of injection training materials, or legal asks for a translation of another party's contract, the document may be full of imperative sentences. The key distinction is between "a request to process content" and "executing instructions inside that content." The user asked for a translation, not for the document's instructions to be followed.

Reflect this distinction in blocking criteria. Rather than blocking based only on document content, use "does processing this document produce a tool call or external address?" to reduce false positives. Controls with frequent false positives eventually get disabled or bypassed, so using inaccurate detection for flagging and human review rather than blocking is itself a security decision.

Injection across multiple turns and memory

Defenses that inspect only one request are weak against attacks spread over time. As conversations grow, assumptions planted in earlier turns can alter later decisions; instructions may also be split into pieces that each look harmless. OWASP LLM01 includes payload splitting, in which instructions scattered through an application distort assessment results. The injection prevention cheat sheet separately classifies delayed attacks spanning interactions or sessions.

Agents with memory extend the duration further. If summarization strips source information, a sentence from an external document may be stored as "a preference the user stated earlier." Three measures address this: save provenance when writing memory, never promote external content to instructions, and periodically clear state through session closure and memory expiration. Evaluation also needs separate multi-turn and multi-session scenarios; single-turn scenarios alone are insufficient (Chapter 4).

The trap in human confirmation: approval fatigue and trust exploitation

Human confirmation is powerful, but it does not work automatically. Someone receiving a hundred approval requests a day will eventually click without reading. If the screen only asks "Execute the tool?" and hides the actual recipient or amount, the person may approve based on the model's plausible summary. The OWASP Agentic Top 10 covers this separately as ASI09 (Human-Agent Trust Exploitation): the agent's confident explanations and authoritative-looking evidence replace independent human verification.

Effective confirmation asks only when risk makes it necessary, shows targets, arguments and effects using original values rather than the model's summary, and asks again if arguments change after approval (Chapter 3). Monitoring approval rates and time to approval can reveal increasing rubber-stamping.

At work

Blue Library: a summary request renews a loan

The public library network Blue Library operates an assistant that summarizes web documents from user-provided links and renews loans on request. One day, a user reports that after reading a blog post for a summary, the assistant renewed a loan without being asked. Investigation finds instructions addressed to the assistant near the bottom of the post, hidden from human view. The impact is small this time, but if the same path had triggered a contact-change tool, it could have led to account takeover.

The team fills in the five-stage table. The attacker could write a public blog (control); the summary feature put its body directly into context (entry); the assistant made a plan unrelated to summarization (deviation); the plan reached the renewal tool without confirmation (propagation); and an actual renewal occurred (impact). None of the five stages was broken.

There are three options. (a) Strengthen the system prompt: almost free, but the incident could recur. (b) Remove web document summarization: most reliable, but it removes the most-used feature. (c) Change the structure. The team chooses (c), adding (a) only as a supporting measure.

1. Web content is passed only as a tool result, labeled "external web page body, author unverified."
2. A lightweight classification model screens the content; suspected instruction insertion produces a filtered summary with a warning rather than the ordinary summary.
3. Loan and account tools are made entirely unavailable in the summarization flow.
4. Renewal executes only after the user confirms the book and new due date on screen and presses a button.

The core assumption is that "injection will eventually pass screening." Thanks to steps 3 and 4, an instruction that passes cannot become an actual action. The team adds 20 fake test documents modeled on this incident's document structure to the regression evaluation set.

Hanmaek Translation: a legitimate document full of attack language

Hanmaek Translation, a business document translation provider, builds a pipeline that translates customer uploads with an LLM, runs automated quality checks and emails the result to the customer. Initially, one agent handles translation, checking and sending-tool calls; it reads the recipient address from the document cover.

Pre-launch testing reveals two problems. First, the red team hides an instruction to "send the translation to another address too" in a fake contract's footnote, and the agent attempts to send to the second address. Second, adding an injection classifier blocks large numbers of customers' security training materials and equipment manuals. Such documents naturally contain imperative sentences like "ignore the previous setting and reset." In blocking mode, the classifier would stop a substantial share of legitimate orders.

The team considers lowering the classifier threshold and having people inspect every document containing imperatives. The first reduces false positives but also lets attack documents through; the second is rejected because it would multiply processing time. Instead, the team restructures the pipeline to break the stage where damage occurs.

1. All tools are removed from the translation model. It reads the original and returns only a translation.
2. Deterministic code handles sending, taking recipient addresses only from the customer account, never the document.
3. Quality-checking code looks for URLs, email addresses or phone numbers absent from the original, and large changes in paragraph count.
4. The classifier switches from blocking to flagging, placing suspicious documents on the quality lead's review list.

Now, even if the translation model follows instructions in the document, it can only produce an unusual translation; step 3 and human review expose that problem. Security training materials are no longer blocked. The team records "subtle distortion in translations" as residual risk, requiring human review for documents with significant legal effect, such as contracts.

Common mistakes

1. "A jailbreak filter stops injection too." A jailbreak filter checks the user input path. Instructions arriving through documents, web pages or tool results bypass it and need separate separation and screening.
2. "Adding retrieved evidence reduces injection concerns." Evidence improves accuracy, but the added document itself creates a new path. Research cited by OWASP also finds that retrieval augmentation and fine-tuning alone do not fully mitigate the vulnerability.
3. "Just blocklist known attack phrases." Small wording changes bypass the list, while legitimate documents get blocked. Pattern filters are only one of several layers.
4. "Putting external documents in the system prompt helps the model understand them." This does the opposite of what is needed, placing untrusted text in the most authoritative position. Pass external content as tool results.
5. "Temperature 0 prevents variation." Anthropic's API documentation says results are not fully deterministic even at temperature 0. Even if answers repeat, they may simply repeat compliance with hidden instructions. The OWASP cheat sheet also notes that temperature 0 did not eliminate jailbreaks.
6. "We wrote the internal wiki, so we can trust it." Authors determine trust level, not storage location. A store that outsiders can edit or into which external material can be pasted is an external path.
7. "A person approves it, so it is safe." Approval becomes a formality if the screen hides actual arguments or requests are too frequent. Limit what requires confirmation and show original values.
8. "Without leakage, injection does no damage." Answer contamination distorts decisions without leakage. Recommendation, summary and assessment features need evidence display and verification.

Summary at a glance

The day before the exam, read each sentence aloud to check your understanding.

• Prompt injection can occur through input invisible to people; its root cause is that instructions and data share one text stream.
• Direct versus indirect describes the path; jailbreak describes the goal of bypassing safety policies.
• Indirect paths include everything the model reads on the user's behalf, including tool results and definitions.
• Analyze paths in five stages: control → entry → deviation → propagation → impact.
• Damage has three branches—action hijacking, exfiltration and answer contamination—with different controls.
• Separation, labeling, structuring and screening reduce probability; privilege separation, confirmation and output restrictions contain damage.
• Put developer instructions in the user turn after tool results, not inside them.
• Evaluate classifiers per language and define behavior when no judgment is available.
• Fail each layer in turn and check that the next layer supports it.

Check the official docs

• OWASP LLM01:2025 Prompt Injection: definition, relationship to jailbreaks, mitigation strategies and scenarios.
• Anthropic — Mitigate jailbreaks and prompt injections: mitigations by threat model, tool_result handling and screening.
• OWASP LLM Prompt Injection Prevention Cheat Sheet: attack types and limitations of each defense.
• OpenAI — Safety best practices: input limits, output token limits and human review.
• Microsoft — Prompt Shields: detection of the two attack types, performance by language and Spotlighting.
• MCP specification — Tools: human involvement and trust in tool annotations.
• Anthropic — Computer use tool: preventing injection through screen and web content.

Self-check

2-1. In one sentence, state how document review must change given OWASP's point that input invisible to people can cause injection as long as the model reads it.

2-2. Jailbreak and indirect injection describe different dimensions. State the basis of each distinction and invent one example with both properties.

2-3. Give one design that breaks the "deviation" stage and one that breaks the "propagation" stage of the five-stage attack chain.

2-4. Which kind of attempt becomes harder when third-party text is wrapped as a JSON string value? Also state one remaining risk.

2-5. State two things to do before applying a document attack classifier trained mainly on English to Korean customer documents.

2-6. Explain why answer contamination does not trigger leakage-detection alerts, and give one control to reduce it.

2-7. Give two reasons human confirmation can become a formality despite an approval screen, and a design measure for each.

2-8. Design task: an agent reads public product review pages, summarizes pros and cons, and posts the result to an internal wiki. Propose a three-line defense combination with one probability-reducing layer and two damage-containing layers.

Hands-on tasks