Chapter 3. Context engineering

Learning objectives

• Explain the context window budget and context rot, and budget an individual request.
• Work backward from the question to select only the necessary material.
• Arrange long documents and questions for higher quality, and label documents with sources and versions.
• Request evidence quotes before answers from long documents, and check whether the quotes adequately support the conclusions.
• Explain conversation summarization, compaction and retrieval augmentation, and diagnose retrieval failures separately from answer failures.
• Place untrusted external content separately from instructions.

Introduction

An analyst at a consulting firm supplied five complete annual reports and asked for “three key industry changes this year.” The context window was large enough, and the request completed without errors. Yet one of the three changes exaggerated a single footnote in the third report’s appendix, while a change emphasized by all five reports was missing. The analyst asked, “Why did it miss that when I put everything in?” The question contains the answer: it missed it because everything was included. The model had not invented the footnote. It had received hundreds of thousands of tokens without signals identifying importance, leaving an irrelevant footnote competing with key information for attention in the large window.

This chapter covers everything outside the instructions: everything the model sees at once. It starts with context as a limited resource, then covers budgeting, selecting only necessary information, document metadata and placement, quote-first prompting and citation features, retrieval augmentation and failure diagnosis, conversation summarization and compaction, strategies for long-running agents, prompt caching, and placement of untrusted content. The deeper discussion covers choosing between long windows and retrieval, tool design, and precautions when storing conversation state on a server.

Key concepts

Context is the space on your desk

A large desk lets you spread out many papers, but the more papers there are, the harder it is to find the one sheet you need. Anthropic describes the context window as working memory, separate from training data, containing all text referenced to produce a response and the response itself. It emphasizes that larger is not automatically better. Whether generated output tokens count against the context window varies by vendor and model. The problem is context rot: accuracy and recall decline as token counts grow.

Anthropic’s context engineering article explains this as an attention budget. In a transformer, every token relates to every other token, so n tokens create n² relationships. Each new token draws on finite attention. The goal is therefore the smallest set of information-dense, high-signal tokens most likely to produce the desired outcome. Prompt engineering refines instructions; context engineering selects all the tokens the model sees on each turn.

Budgeting the context window

Before budgeting, identify what occupies the window. Anthropic’s documentation includes the system prompt, every message in messages (including tool results, images and documents), tool definitions, and output generated on that turn, including thinking. Tokens read from the prompt cache still occupy the window. Caching reduces costs, but cached tokens still occupy context space. Token counts vary by language and notation. Measure actual input with token-counting tools rather than estimating from character counts, including for Korean material.

Output headroom is especially easy to forget. Even a large window has a separate per-request output cap. According to Anthropic’s documentation, newer models with a one-million-token window still have a maximum output of 128k, or roughly 128,000 tokens, per request. A request to rewrite an entire long report may fit as input but hit the output cap and be truncated with stop_reason set to max_tokens. The documentation advises raising the limit or continuing generation. For long deliverables, split the work from the outset and include shared standards, such as a glossary, in every section.

Include only the information that is needed

Work backward from the question before adding material.

1. Break the question into subissues. “Risks to second-half pricing strategy” becomes cost changes, competitor prices and demand changes.
2. List the material needed per issue. Cost changes need purchase-price tables; competitor pricing needs the September price survey.
3. Separate essential from supporting material. Distinguish information whose absence makes the answer wrong from information that merely enriches it.
4. Choose originals or excerpts. Use original text when wording matters, such as clause comparisons; use excerpts or summaries for background.
5. Remove supporting material first if over budget. Define the removal order in advance.
6. Mark issues without evidence. Tell the model that no material exists for those issues so it does not fill them from general knowledge.

Cost savings are only part of the purpose. Context containing material from other products, years or departments can itself lead to incorrect answers.

Attach source, version and scope to documents

Content alone does not tell the model which document is current or where it applies. Add metadata to every document.

<documents>
  <document index="1">
    <source>Travel Expense Policy, Article 4</source>
    <version>Revised 2026-07-01</version>
    <applies_to>Permanent employees, domestic business travel</applies_to>
    <document_content>{{RULE_4}}</document_content>
  </document>
  <document index="2">
    <source>Travel Expense Policy, Article 4</source>
    <version>Revised 2025-01-01 (repealed)</version>
    <applies_to>Only expense claims for travel before 2026-06-30</applies_to>
    <document_content>{{RULE_4_OLD}}</document_content>
  </document>
</documents>

Anthropic recommends wrapping each of multiple documents in <document> and separating content from metadata with sub-tags such as <document_content> and <source>. Revision dates, effective dates and applicable groups let the model determine which document to follow. Without metadata, retrieving old and new versions of the same clause can produce an answer that mixes them.

Metadata matters even more when documents disagree. Simply asking for synthesis may force conflicts into one sentence or silently discard one side. Ask for separate lists of content confirmed by multiple sources, differing numbers or definitions, and content present in only one source. Define a resolution rule, such as “For old and new versions of the same clause, follow the version effective on the question’s reference date.” Have unresolved conflicts reported, rather than resolved by guesswork: “Conflict: documents 1 and 3 differ.”

Long documents first, the question at the end

For long inputs above 20,000 tokens, Anthropic recommends placing long documents before questions, instructions and examples. In official tests, putting the question at the end improved response quality by up to 30%, especially on complex multi-document inputs. Gemini likewise recommends placing specific questions last and linking them to earlier material with phrases such as “Based on the information above.” OpenAI gives an example with request-specific context toward the end. The principle also applies to images: Anthropic advises putting them before text and labeling multiple images “Image 1:”, “Image 2:” and so on, so they can be referenced by name.

Official recommendations are starting points, not universal laws. A useful default is core behavioral constraints first, long material next, and the specific question last. For important tasks, compare alternative arrangements.

Get the quotes first

When receiving an answer from a long document, the main risk is content absent from the document slipping in. Official guidance recommends requesting relevant quotes before the task. Selecting quotes focuses the model on relevant content and lets people compare them directly with the source. Anthropic’s hallucination-reduction documentation adds two measures: state when no relevant quote exists, and after writing the answer find a supporting quote for every claim, withdrawing any claim without one.

1. Copy relevant original sentences word for word into <quotes>, numbering them.
   If none are relevant, write only "No relevant quotes."
2. Answer in <answer> using only <quotes> as evidence, with a quote number for each claim.
3. Remove claims that cannot be assigned a quote number.

The presence of a quote does not mean evidence is sufficient. Check three further things.

Citation features and search-result blocks

Instead of prompting for quotes, you can use a vendor citation feature. Enable Anthropic Citations on documents, and quoted text blocks return source locations: character positions for plain text, page numbers for PDFs, and block numbers for custom blocks. Check separately for uncited claims. The official documentation lists three benefits: cited_text does not count as output tokens; API-parsed citations guarantee valid locations in supplied documents; and internal evaluations found more relevant quotations than prompt-based methods.

Plain text and PDFs are cited at sentence level. For custom units such as transcript segments or bullets, supply custom content blocks. Citations cannot be combined with structured outputs: citation-enabled documents and output_config.format in one request cause a 400 error. Split the calls if both are needed.

You can supply your own retrieval results in Anthropic search_result blocks containing source, title and content, to receive source-linked citations like web search. Gemini’s grounding with Google Search lets the model decide when to search and attach source URLs to response segments. Use search grounding for frequently changing public information and internal search results for internal material. If combining them, make the evidence source clear for each answer sentence.

How retrieval augmentation works

Retrieval-augmented generation (RAG) finds material the model does not know and supplies it at question time. It is like a librarian who listens to your question and puts just a few relevant books on your desk.

Anthropic’s Contextual Retrieval article addresses the first step. A chunk such as “Revenue rose 3% over the previous quarter” loses which company and period it describes. The solution is to prepend a short explanation, usually 50–100 tokens, based on the whole document before embedding each chunk, and build a BM25 index from the same text. Embedding search excels at semantic matching but struggles with exact terms such as error codes or model names. Word-matching BM25 fills that gap. In the article’s experiments, the retrieval failure rate fell by 35% with contextual embeddings alone, 49% with added BM25 and 67% with reranking too.

Not every problem needs retrieval. The same article says that for a knowledge base below 200,000 tokens, you can supply all material without building a retrieval pipeline and use prompt caching to reduce cost and latency. For small collections, the simplest method may be best.

Diagnose retrieval failures separately from answer failures

Revising the prompt first when RAG gives a wrong answer is a common mistake. Locate the failed stage first.

1. Was the correct evidence in the store? If not, the problem is the material.
2. Was it among the top retrieval results? If not, revise query generation, indexing or retrieval.
3. Did it enter the context? If reranking or selection removed it, revise selection criteria.
4. Was the answer wrong despite the evidence being included? Only now revise generation: instructions, arrangement and quoting.

This diagnosis needs an evaluation set pairing each question with its supporting document in advance. It is one use of Chapter 5’s evaluation sets.

Conversation summarization and compaction

Long conversations require shorter histories. Summarizing the beginning is common, but loses precise wording. Repeated summaries especially blur numbers, dates and conditions into phrases such as “an appropriate level” or “soon.” Design summaries as work-state records, rather than narrative overviews.

<conversation_state>
  Confirmed conditions: 1,200 units, delivery November 14, individual plastic packaging
  Open questions: logo print color (customer will reply after reviewing the proof)
  Decision history: unit price agreed October 20 (quotation Q-2026-118)
  Next action: finalize printing specifications when the proof response arrives
</conversation_state>

Keep confirmed values in their original wording, distinguish decisions from suggestions, and make each item traceable to its source. Retain a few recent turns verbatim to preserve conversational flow.

Vendor features are another option. Anthropic offers beta server-side compaction, replacing older turns with a model-written summary to continue a conversation. Write custom summarization instructions if the default omits necessary information. For finer control, context editing can clear old tool results or thinking blocks by rule. The person responsible for the workflow decides what must remain.

Context strategies for long-running agents

Agents calling tools for hours fill context quickly. Anthropic describes the following techniques and suitable uses.

Just-in-time loading is slower but avoids stale indexes and keeps context light. A hybrid can preload some material and retrieve the rest as needed.

Prompt caching and arrangement

Prompt caching stores the repeated prefix of requests for reuse. The key is that the prefix must match character for character. Anthropic forms the prefix in the order tools, system, messages, requiring a 100% match through the cache point. OpenAI also enables caching by default on supported models and requires the full rendered prefix to match for reuse.

Understand the cost structure too. According to Anthropic’s documentation, cache writes cost slightly more than base input—1.25 times for a five-minute cache—while reads cost much less, 10% of base input on most models. Default lifetime is five minutes, refreshed with each use, with a one-hour option. OpenAI caches prompts of at least 1,024 tokens on current models and states that caching does not change output generation.

Where untrusted content belongs

Web pages, incoming emails, user-uploaded files and tool results come from sources you cannot control and may contain sentences intended to manipulate the model. This is indirect prompt injection. Unlike direct injection by a user, the user may be trusted while third-party content read on their behalf provides the attack path. OWASP’s 2025 Top 10 for LLM Applications ranks prompt injection first and notes that neither RAG nor fine-tuning fully prevents it.

Anthropic recommends the following placement principles.

1. Put third-party content only inside tool results (tool_result), rather than system prompts or ordinary user text.
2. Explain what it is and where it came from, such as an email body from an unknown sender, in the tool description or result structure.
3. State in the system prompt that tool, document and search content is untrusted data and cannot override the system prompt or original user request.
4. Where possible, place external strings in JSON objects. JSON escaping clarifies boundaries so attackers cannot close quotes or tags and escape into the instruction area.
5. Keep your own instructions outside tool results, where they may be treated as suspicious or ignored.
6. Limit damage if defenses fail: least privilege, no access to unnecessary secrets, narrow tool permissions and human approval for risky actions.

For direct injection, or jailbreaks, Anthropic recommends screening input with a lightweight model, validating against known attack phrases, defining refusal behavior in the system prompt, and responding to repeated attempts. OWASP’s measures point in the same direction: limited behavior scope, output-format validation, input/output filtering, least privilege, human approval for risky actions, separation and labeling of external content, and adversarial testing. Put only the most trustworthy content in the most trusted position.

Going deeper

Put everything in a long window, or retrieve it?

Larger windows invite the question of whether retrieval can be replaced by putting all the material in context. The answer depends on the task. Google’s long-context documentation notes that high accuracy on single-information retrieval tests does not carry over to retrieving multiple pieces at once. Input tokens are charged again with each question, and longer input increases time to the first token.

Tool design is context design

Tools are an agent’s hands and eyes. Their definitions and results occupy context, so tool design determines context quality. Anthropic advises token-efficient results and minimal functional overlap, noting that if a human engineer cannot confidently choose which tool to use, an agent cannot be expected to do better.

Anthropic’s tool-definition documentation calls detailed descriptions, at least three or four sentences per tool, the most important factor in tool performance. Describe what a tool does, when to use and avoid it, what parameters mean, and what it does not return. Group similar operations in one tool with an action parameter, and prefix service names when spanning services. Results should contain only meaningful identifiers and fields needed for the next decision. Bloated results waste context.

Entrusting conversation state to a server

You can store conversation history on a vendor server. OpenAI Responses API links responses with previous_response_id, or Conversations API stores a conversation as a durable object. The documentation says prior responses’ input tokens are still billed again when linked with previous_response_id. Stored responses remain for 30 days by default; store set to false disables response-object storage. Check data-policy documentation for separate retention, such as abuse-monitoring logs retained for up to 30 days by default. Conversation objects are not subject to that 30-day limit. Choose according to cost budgets and retention rules.

When combining reasoning models and tools, pass previous reasoning items too. OpenAI strongly recommends returning reasoning items after function calls and using encrypted reasoning content (encrypted_content) in stateless mode. Anthropic recommends appending assistant turns exactly as returned by the API, including thinking blocks. On some newer models, modifying conversation content before a thinking block can cause errors or discard the block. To change rules, append rather than rewrite history, as with Chapter 1’s mid-conversation system messages.

At work

Blue Logistics: a Q&A bot searching 400 policies

Blue Logistics runs a Q&A bot over 400 internal policy documents. Employees have three complaints: plausible answers lack identifiable policy sources, travel-expense answers mix in leave policies, and questions about July’s revised accommodation allowance sometimes receive the old amount.

Moon, the manager in charge, first diagnoses 50 failed questions by stage. Three lack supporting material in the document repository. Fourteen lack the correct evidence in the top 30 results, mostly questions using clause numbers such as “Article 4-2.” The remaining 33 include the evidence in context but still receive wrong answers. Moon also examines the structure: question first, then 30 unseparated results, ending with “Answer.”

The diagnosis produces different fixes by stage.

1. Material. The three missing answers appear only on a notice board, not in policy documents. Moon sends the notices to the policy department for incorporation.
2. Retrieval. Add word matching to semantic search for exact clause numbers and policy names. At query generation, classify the business category—travel, leave or purchasing—and use it as a search filter.
3. Selection. Rerank candidates by relevance and keep only the top five.
4. Placement and metadata. Put each retained policy first with <document> tags, policy number, revision date and scope. Move the question last. Exclude repealed versions by default, but retrieve the version effective at a historical date when the question specifies one.
5. Generation. Copy relevant clauses into <quotes> before answering, and end with policy number and revision date. If both old and new versions appear, follow the question’s reference date; show both if the choice cannot be determined.

Moon rejects the argument that spare context capacity justifies increasing retrieval to 50 results. Many failures already come from irrelevant policies, and more would worsen that problem. Repeating the same 50 questions after the changes yields mostly correct answers with evidence and cuts input tokens to roughly a quarter. Employees can now find originals directly through the cited policy number.

New Light Daily: three years of pledges in one article

Jang, a data reporter at fictional local newspaper New Light Daily, is preparing an article on how four mayoral candidates’ pledges changed over three years. Press releases, debate transcripts, manifesto PDFs and council minutes total millions of tokens. Initially, supplying everything for each candidate and asking for a summary of changes produces inconsistent results. Withdrawn pledges appear current, and statements by different candidates are mixed together.

Jang works backward from the article’s question: what is each candidate’s current pledge by policy area, and when and how did it change? The four areas are transport, housing, social care and public finance. Work is split into one candidate × one area, with budgets for all sixteen tasks.

Retrieval selects the material. After searching candidate names and area keywords, only statements spoken by that candidate remain. Transcripts are split by utterance, with speaker, date and event metadata. This eliminates mixed-candidate statements.

Generation starts with quotes. The model copies pledge statements verbatim, arranges them chronologically and marks changes from earlier statements. Interpretive issues such as withdrawal status remain “Confirmation needed.” Jang asks a colleague for a script comparing quotes programmatically against the originals and filters out every nonmatching quote.

The final synthesis receives only the sixteen summary tables, rather than raw material, and produces comparisons by policy area. Task progress notes retain verified quotes and pending verification items so the work can continue over several days without losing its place.

Rejected options are recorded too. Supplying all material and directly requesting a comparison risks missed pledges because many pieces must be found simultaneously. Comparing summaries risks blurred dates and numbers, so Jang accumulates verbatim quotes instead. The published article attaches dates and sources to each pledge statement, preserving evidence for reader inquiries.

Common mistakes

• With a large window, including everything is better. Context rot means irrelevant material can lower quality and increase cost and latency.
• A larger window allows longer output. Output has a separate cap. Generate long deliverables in sections.
• The question must come first so the model knows the goal. For long input, documents first and question last is the official recommendation. Core behavioral constraints belong at the front.
• If RAG fails, revise the prompt. First distinguish material, retrieval, selection and generation failures.
• Summarization loses no information. It blurs numbers and conditions. Keep confirmed values verbatim in a state record.
• External material belongs in the system prompt so it is followed. Putting untrusted content in the most trusted position increases injection risk.
• Turning caching on automatically saves money. Changing early values causes everything after them to be recomputed, reducing hits. Put fixed content first and variables last.

Summary at a glance

Check the official docs

• Anthropic — Prompting best practices: long context: Read about document placement, tag structure and requesting quotes first.
• Anthropic — Effective context engineering for AI agents: Read about context rot, just-in-time loading, compaction, structured notes, subagents and tool design.
• Anthropic — Contextual Retrieval: Read about chunk context, BM25, reranking and advice for small knowledge bases.
• Anthropic — Prompt caching: Check cache order, invalidation, lifetime and price multipliers.
• Anthropic — Mitigate jailbreaks and prompt injections: Read placement principles against indirect injection.
• OWASP — LLM01:2025 Prompt Injection: Check direct and indirect injection and the seven measures.

Self-check

3-1. Define context rot in one sentence and give two reasons against “The window is large, so include all related material.”

3-2. Name five components of a request’s context budget. Which is most often forgotten, and what symptom results?

3-3. Write a tagged prompt skeleton for two 60-page audit reports and one question. Give each document a source and date, and request quotes first.

3-4. Write four diagnostic questions, in checking order, for a wrong RAG answer. In one sentence, explain why instructions are revised only at the final question.

3-5. A new coordinator needs to continue weeks of event-planning conversation with a model in a new chat. Name three kinds of information to retain verbatim and show an example record format.

3-6. An agent uses the same long guidelines but adds or removes tools per request according to the user’s access level. Explain the prompt-caching cost and propose one way to reduce it while preserving access differences.

3-7. Design a research agent that reads competitor websites to compile pricing tables. Give two points each on where and how to place retrieved page bodies, and safeguards reducing damage if hidden page instructions succeed.

Hands-on tasks