Chapter 3. Context, memory and state

Learning objectives

• Explain context rot and attention budget, and establish criteria for choosing information to put in context.
• Choose compaction, tool result clearing, structured notes or subagents according to the task.
• Distinguish conversation history, task state, long-term memory and systems of record, and prevent context pollution during long runs.
• Design resumption with checkpoints and idempotency, checking external state when resuming.
• Structure work spanning several sessions so the next executor can take over.

Introduction

On the afternoon of its second day on a two-day task, an agent says something strange: “The work is almost finished.” Inspection shows half the work untouched, and a rule agreed with the team on the first morning has vanished without a trace. The agent is not being lazy. Context has filled several times and been replaced with summaries. The summaries preserved continuity but not exact state.

For a person, this would be like running a long project entirely from memory, without notes. This chapter explains how to give an agent a notebook and a work journal. It covers what belongs in context and outside it, choosing compaction, external memory and subagents, and preventing pollution, then moves on to checkpoints, resumption and handoff.

Key concepts

Context is a finite budget

Anthropic defines context engineering as the strategies for curating and maintaining the optimal set of tokens entering the model during inference. Prompt engineering refines instructions; context engineering covers everything the model sees, including system prompts, tool definitions, tool results, message history and external data.

The rationale is context rot: the observation that as context tokens increase, the ability to recall information accurately from them declines. The article explains that LLMs have an attention budget. Transformers compute relationships between each token and other context tokens. The number of relationships grows proportionally to the square of the token count (n²), spreading attention thinner as tokens increase. Every new token consumes some of this budget.

Context is like space on a desk. Even a large desk makes finding the one page you need difficult if all the papers are spread across it. A larger window means you can put more in, not that everything inside is used equally well.

What to include: altitude, tools and examples

The article gives criteria for each context component.

The “right altitude” is an aircraft analogy. Fly too low and every terrain feature becomes an obstacle; fly too high and the destination is invisible. A prompt listing 180 exceptional cases as rules breaks other cases whenever one rule changes. Principles and representative examples, with expected behavior protected by an evaluation set (Chapter 4), are easier to maintain.

Just-in-time retrieval and progressive disclosure

Instead of loading all data in advance, an agent can retain only lightweight identifiers—file paths, saved queries or links—and retrieve data through tools when needed. This is just-in-time retrieval. Claude Code inspects needed portions using commands such as head and tail, rather than reading large datasets in full. Discovering relevant context gradually during exploration is called progressive disclosure. It resembles using a contents page and bookmarks instead of memorizing an entire book.

Just-in-time retrieval does not always win: searching at runtime is slower than preloading. The article therefore introduces a hybrid strategy. Claude Code loads CLAUDE.md initially and finds other files as needed through glob and grep. Preload stable material needed almost every time; retrieve large, frequently changing material. Domains with less dynamic context, such as law and finance, can preload more.

Successful retrieval does not mean the correct answer has been found. Results should carry source trustworthiness, the document's applicability date and the current user's access permissions. Retrieval design also includes saying “I did not find it” rather than declaring “it does not exist” when evidence is not found.

Four layers of information

Long-running systems contain four kinds of information. Mixing them causes incidents.

A system of record is the authoritative system for business facts. An agent may remember processing a refund, but the payment system's ledger determines whether it actually occurred. Task state, rather than conversation history, should determine the next action on resumption. If task state conflicts with the system of record, the system of record prevails.

Compaction and tool result clearing

Compaction summarizes a conversation as the window approaches its limit and starts a new window with that summary. Anthropic explains that good compaction preserves architecture decisions, unresolved bugs and implementation details while discarding redundant tool output. It recommends tuning compaction prompts first for completeness (recall), then for reduced clutter (precision).

The lightest and safest compaction is tool result clearing, because old raw tool results deep in a conversation usually need not be read again. Platform features can help. Claude API context editing offers server-side strategies for clearing old tool results, with controls for when to start (trigger), how many recent results to retain (keep), a minimum amount to clear at once (clear_at_least), and excluded tools (exclude_tools). Claude API also offers server-side compaction in beta. OpenAI Agents SDK has a session implementation that compacts session history through the Responses API. Names and options can change; check the documentation.

Structured notes and external memory

With structured notes, agents write notes outside their context window and read them later. Anthropic's examples include Claude Code to-do lists, NOTES.md files, and a Pokémon experiment recording progress and maps over thousands of steps. This fits iterative work with clear milestones.

Memory tools turn this into a product feature. Claude's memory tool runs on the client side. The model requests file operations under /memories; the application's handler executes them in its chosen storage. Safeguards are therefore the application's responsibility. Documentation recommends normalizing every path and verifying that it stays within /memories to prevent traversal such as ../, filtering sensitive information, limiting file sizes and deleting long-unused files.

Google ADK distinguishes two layers. Sessions and state provide short-term memory for the current conversation; a memory service stores long-term memory retrieved across past conversations. Add with add_session_to_memory and retrieve with search_memory. Development in-memory implementations disappear on restart, so use a persistent implementation in production.

Isolating context with subagents

In a subagent architecture, specialized subagents perform focused work in clean windows and return only condensed results. Anthropic says that even after exploring tens of thousands of tokens, subagents typically return summaries of 1,000–2,000 tokens. Exploration noise stays out of the main agent's window, letting it focus on synthesis and judgment. This suits complex investigations benefiting from parallel exploration.

Some outputs cannot be summarized. If the main agent transcribes a table or code that must remain exact, it fills context and introduces transcription errors. Anthropic's research system reduced information loss and wasted tokens by having subagents write results to external storage, such as files, and return only lightweight references, such as paths or storage keys. The article also describes the lead agent saving plans in memory because context is truncated beyond 200,000 tokens.

The three techniques can be compared as follows.

Preventing context pollution

Context pollution occurs when incorrect, outdated or untrusted information accumulates in context and repeatedly distorts subsequent decisions. It has three sources: traces of failed attempts, where repeated failures anchor the model to the same path; untrusted content, such as instructions in web pages or documents; and stale memories of facts that have changed.

Use these rules to prevent it.

1. Clear raw tool results that have served their purpose, retaining only necessary points and references.
2. When the same failure repeats, condense its history into one summary line, explicitly choose a new approach and restart. Use a clean subagent if needed.
3. Mark external content with its source and distinguish it from instructions. Do not obey instructions within it (Chapter 5).
4. Attach sources and observation times to memories; reverify or discard old entries.
5. Do not promote user statements to system-level instructions without verification. Memory is data, not authority.

Prompt caching and context layout

Long-running agents resend similar prefixes dozens or hundreds of times. Prompt caching reuses processing of these repeated prefixes to reduce cost and latency. According to Claude API documentation, the cached prefix is built in the order tools, system, messages. Changing one layer invalidates that layer and all later ones. Changing tool definitions invalidates the entire cache. The default lifetime is five minutes, refreshed on use, with a one-hour option available. Each model has a minimum cacheable length; shorter prefixes are not cached even if marked. Check hits through cache_read_input_tokens and cache_creation_input_tokens in responses.

Common mistakes include putting per-request values such as the current time or request number before fixed instructions, recreating tool definitions with changed contents or order, and using JSON serialization with randomized key order. Put stable content first and changing content later. Tool result clearing also conflicts with caching: the cache is invalidated from the deletion point. Documentation therefore recommends clearing enough at once (clear_at_least) to make the invalidation worthwhile.

Checkpoints and idempotency

Long-running work with external effects should not restart from the beginning after failure. Anthropic's research system article notes that agents are stateful and errors accumulate. It describes combining systems that resume at the point of failure with deterministic safeguards such as retry logic and regular checkpoints. It also recommends informing the agent when tools are failing, because even small failures can be fatal.

A checkpoint is a durably stored stage result that supports resumption. An idempotency key is an identifier ensuring a repeated request produces its effect only once. The design has three elements.

1. Record what each stage did, when and with what result in durable storage.
2. Attach an idempotency key derived deterministically from the task and target to external calls, for example taskID:targetID:stage.
3. On resumption, determine the next action from stored records and the system of record, rather than the model's memory.

Designing resumption after interruption

Continuing from a saved point does not by itself make resumption safe. The world may have changed since saving. Include these checks.

Frameworks support this too. OpenAI Agents SDK can serialize a run awaiting human approval as RunState, store it in a database or queue, then restore and resume it. Documentation recommends server-controlled storage and restoring only trusted snapshots, since serialized state includes approval decisions, pending tool calls and their arguments. Store agent definition or SDK version markers with the state. In Google ADK, state updates must be recorded through event state_delta to persist. Directly modifying session.state on a session fetched from a service may not be tracked or saved. State values must use serializable basic types.

Handoff for long-running work

Anthropic's long-running coding agent experiment observed two failures: trying to do too much at once and exhausting context mid-task, and later sessions seeing partial progress and declaring the work complete. The solution separated roles.

• Initializer agent (first session): creates init.sh to start the development server, claude-progress.txt to record agent work, an initial commit showing the first added files, and a feature list with more than 200 features all marked “failing.”
• Coding agent (later sessions): advances only one feature at a time, leaves descriptive commits and progress records for the next session, and ends with the environment in a clean state.

The feature list used JSON because the model was less likely to edit or overwrite it inappropriately than Markdown. Coding agents could change only passes fields, and deleting or altering tests was explicitly prohibited. Each new session checked the working directory, read Git history and progress files, selected the highest-priority unfinished feature, started the development server and tested basic behavior before beginning work. Features were marked passing only after human-like end-to-end testing.

Generalizing this produces a handoff package: documentation enabling the next executor to continue without reading the entire conversation. Include the goal and completion criteria; completed items and their verification; remaining work and priorities; confirmed facts versus assumptions; artifacts and external operation identifiers; and prerequisites for the next action and points requiring additional approval. The key is verified artifacts rather than self-assessments such as “almost done.”

Where to keep conversation state

Decide who manages conversation history. OpenAI Agents SDK offers three approaches: manual management, passing results directly into the next input list; sessions, loading history before a run and saving new items afterward; and server-side conversation management through conversation_id or previous_response_id. Documentation advises choosing one persistence strategy per conversation. Sessions cannot be combined with server-side continuation options in the same run, and mixing client-managed history with server-managed state can duplicate context.

Google ADK state uses key prefixes to determine scope.

Choosing the wrong scope can leak one user's information to another or lose needed information at the next session.

Going deeper

The gap between checkpoints and external side effects

Checkpoint storage and external system changes do not share one transaction. A process can die immediately after a successful external call but before recording its result. Resumption code reading only the record concludes “not done yet” and performs the action again.

The standard procedure is record intent → execute → record result. Before executing, record the intended action and idempotency key. Execute, then record the result. On resumption, if intent exists without a result, query the external system using the key before re-executing. If processed, record only the result. If not, execute with the same key. If the external system does not support idempotency keys, require human confirmation especially where duplicates are costly. If some steps cannot be reversed, design compensating actions as discussed in Chapter 5.

The memory lifecycle

One incorrect long-term memory can repeatedly affect many tasks. Memory therefore needs a lifecycle.

1. Promotion criteria: Decide what becomes memory. Retain repeated or user-confirmed information rather than one-off remarks.
2. Source and time: Store who said it, when and how it was verified.
3. Conflict resolution: When new information conflicts with old memory, resolve it by recency and source authority, preserving change history.
4. Expiry and reverification: Set a lifetime; reverify or discard expired information.
5. Propagating correction and deletion: When users request deletion, remove not only originals but search indexes and summaries derived from that memory.

The fifth stage intersects with privacy obligations. Set retention and deletion criteria with the privacy officer according to applicable laws and terms. Technically, isolate memory storage by user and test that deletion actually occurs.

Evaluate memory and compaction policies too

Changing compaction prompts or memory policy may reduce costs while lowering accuracy. Evaluate the policy itself. Compare against a no-memory baseline, add tasks checking adherence to earlier decisions immediately after compaction, and deliberately seed stale memories to see whether they are filtered out. Success rate after forced interruption and resumption is also a useful metric.

At work

Milky Way Books: decisions lost during 1,800-book conversion

The ebook distributor Milky Way Books assigned an agent to convert 1,800 older ebooks into a new format and fill in accessibility information. The first attempt used one long session with automatic compaction. On day three, the agent forgot a rule agreed on day one—take alternative text for table images from the original captions—and processed 140 books differently. Another 60 books were marked complete without running the validator.

The team rejected two alternatives. A model with a larger context window would only postpone decisions disappearing during summarization. Adding “do not forget decisions” to the system prompt repeated the mistake of entrusting memory to the model.

In the new structure, the first session only initializes. It creates and commits books.json with all 1,800 statuses (pending, converted, validated, failed) and validation result paths, decisions.md with agreed conversion rules, and init.sh to run the converter and validator. Later sessions read the three files and Git history at startup and process 20 books at a time. Only validator passes permit changing state to validated. Raw validator logs are removed from context after retaining key points. New rules go into decisions.md and are committed before work continues.

The next 1,000 books needed no rework, and none were marked complete without validation. Context usage per session actually fell.

Furrow Policy Institute: taking over three-day research

Furrow Policy Institute spends three days researching youth housing support policies across local governments to produce a comparison table. The first day's session left a summary: “23 of 41 researched.” The next morning, the client added income eligibility to the comparison criteria, while two researched governments announced changed support amounts. The second session trusted the summary and researched only the remaining 18. The final table retained two outdated amounts and left income eligibility blank for the first 23.

The team compared alternatives. Passing the entire first-day conversation to the next session filled context without revealing changes. Starting research from scratch daily tripled cost. The team chose a handoff package and resumption checks.

The package records the criteria document version, each government's status and researched fields, evidence URLs and verification times, assumptions, and the next session's first action. Before starting, the new session checks the criteria version. If it changed, governments missing newly added fields return to the research queue. Evidence older than 24 hours is checked for announcement page updates first. Results are stored in separate files per government; sessions carry only file paths.

In the next investigation, criteria changed once and announcements changed in five locations, but the final table contained zero outdated values.

Common mistakes

1. “A large window removes the need for context management.” A large number suggests everything will fit, but recall accuracy declines as the window fills. A larger window merely postpones the need.
2. “A compaction summary preserves progress state too.” Continuity is reassuring, but exact state and early decisions can disappear. Keep precise information in structured records.
3. “To resume, ask the model how far it got.” Plausible answers are convenient, but a model in a new process does not know the earlier run. Determine state from records and the system of record.
4. “Resuming from a checkpoint is safe.” Saving is reassuring, but external state may have changed or approvals expired. Reconcile before resuming.
5. “A subagent returning an entire table ensures accuracy.” Receiving the original looks accurate, but the lead agent fills its context window and introduces errors while transcribing. Store results and return references.
6. “Putting user statements directly into long-term memory provides personalization.” It is quick, but unverified promotion creates a route for stale information and instruction injection.
7. “The more often tool results are cleared, the better.” Clearing lightens the window, but can invalidate the prompt cache each time. Clear sufficient amounts together.

Summary at a glance

Check the official docs

• Anthropic: Effective context engineering for AI agents — context rot, right altitude, just-in-time retrieval and three long-task techniques
• Anthropic: Effective harnesses for long-running agents — initializer and coding agents, JSON feature lists and session startup
• Claude API: Context editing — tool result clearing options and caching interactions
• Claude API: Memory tool — client-side memory, path traversal prevention, sizes and expiry
• Claude API: Prompt caching — cache ordering, invalidation and lifetimes
• OpenAI Agents SDK: Sessions — session implementations and server-side continuation
• OpenAI Agents SDK: Human-in-the-loop — RunState serialization and storage precautions
• Google ADK: State — state prefixes, persistence and state updates

Self-check

3-1. Explain why context management is still needed with a larger window, using the term “attention budget.”

3-2. Why is clearing old tool results considered the lightest compaction, and what needs care when combining it with prompt caching?

3-3. Among conversation history, task state, long-term memory and systems of record, which should determine the next action on resumption? What if it conflicts with the system of record?

3-4. What failure is a long-running coding harness trying to prevent by using a JSON feature list and allowing only pass-status fields to change?

3-5. Explain why checkpoints follow “record intent → execute → record result,” using a process dying just before result recording as an example.

3-6. Give two reasons to store a source and observation time with each long-term memory item.

3-7. Design a five-line contents outline for handing a three-day investigation to another executor. Describe what belongs in each line in one sentence.

Hands-on tasks