Chapter 1. Agent architecture

Learning objectives

• Explain the difference between workflows and agents, and identify where both approaches coexist in one system.
• Distinguish the conditions and failure modes for the prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer patterns.
• Explain why multi-agent structures increase costs, and assess adoption by cost per successful task.
• Choose appropriate connections between handoff and agent-as-tool, and between LLM and code orchestration.
• Apply a decision process that starts with the simplest structure and adds complexity only when evidence supports it.

Introduction

A team rebuilt customer inquiry handling “with agents.” The model chose tools on its own and handed work to another agent when needed. The demo was excellent. Within a month of operation, however, three complaints appeared: even the same question followed a different path each time, making it hard to reproduce; responses were slower; and the bill was three times the estimate. The logs showed that most inquiries were structured requests with an unchanging processing order. The team was calling a taxi and asking for directions every time it could simply follow a fixed route.

This chapter explains how to avoid that mistake. It distinguishes workflows from agents, identifies tasks that need agents, and examines five workflow patterns and the costs of autonomous and multi-agent systems. It then covers ways to connect agents and a decision process that starts with the simplest structure. The deeper discussion examines hybrid designs and correlated errors.

Key concepts

Workflows and agents

Anthropic's “Building effective agents” distinguishes them as follows. Workflows are systems in which LLMs and tools are coordinated through predefined code paths. Agents are systems in which LLMs dynamically direct their own process and tool use. The article explains that workflows provide predictability and consistency for well-defined tasks, while agents are better when flexibility and model-driven decisions are needed at scale.

If a workflow is a bus on a fixed route, an agent is a taxi that chooses its route after hearing the destination. Calling a taxi for a fixed route increases cost and makes the route harder to predict. Conversely, insisting on a bus when every side street requires a different route will not get passengers to their destination.

This distinction is not a label for the entire system. Within one system, which component determines the next action can differ by stage. Code might control inquiry classification and permission checks, while the model independently leads investigation for some classified inquiries. A design document should say “Which component determines the next action at this stage?” rather than “This system is an agent.”

Tasks that need agents

Choosing an agent means accepting higher latency and cost in exchange for better task performance. Anthropic recommends finding the simplest solution first and adding complexity only when needed. For many applications, optimizing a single LLM call with retrieval and examples is usually enough.

OpenAI's “A practical guide to building agents” gives three criteria for considering agents.

The guide adds that you should first confirm a clear fit with these criteria; otherwise, a deterministic solution may suffice. Deterministic means the same input always follows the same path and produces the same result. When rules are simple and stable, as in validating and uploading files in a fixed format, ordinary code is cheaper and more accurate than an agent.

The augmented LLM and the agent loop

All these structures share the same basic components. Anthropic calls the building block an augmented LLM: an LLM extended with retrieval, tools and memory. OpenAI's guide likewise describes an agent in terms of three elements: model, tools and instructions. Tools fall into three categories: data tools retrieve context and information; action tools act on external systems; and orchestration tools call other agents as tools.

An agent runs these components in a loop. The model chooses the next action, executes a tool and receives the result as input again. Anthropic explains that at each step an agent should obtain actual results from the environment—“ground truth,” in its terminology—such as tool results or code execution results to judge its progress. It can pause at checkpoints or when blocked to obtain human feedback. Errors accumulate if the loop runs only on model guesses instead of these actual results.

Prompt chaining and gates

Prompt chaining divides a task into fixed subtasks, passing one call's output into the next. Code-based checks, or gates, can sit between stages. Anthropic's examples include writing and translating marketing copy, and writing an outline, checking it against criteria, then writing the full text. Use chaining when a task can be cleanly divided into fixed steps.

When designing a chain, first define the data contract between stages. A schema fixes what the preceding stage passes, in what format, and what the next stage assumes.

1. Define an output schema for each stage: fields, units and permitted ranges.
2. Check the schema and business rules in code at the gate.
3. If a check fails, rerun only that stage or hand it to a person.
4. Add up stage latencies and check that they fit the total latency budget.

The typical chaining failure is error propagation. If the first stage misreads a unit, later stages write plausible prose using that number. Gates interrupt this propagation. Each additional stage also adds latency, so weigh the benefit of dividing the task—greater focus per call—against that latency cost.

Routing

Routing classifies input and sends it to specialized downstream processing. Use it when there are distinct categories best handled separately. Anthropic gives examples of classifying customer inquiries by type, and sending easy questions to smaller, faster models and difficult ones to stronger models.

The cost of misclassification matters as much as accuracy in router design. Sending a refund inquiry to general information has different harmful consequences from sending a general inquiry into refund processing. Design these elements together.

Classification can use rules, a small model or an LLM call producing structured output. Code or a small model is cheaper for stable categories. LLM classification is better when category boundaries depend on natural-language nuance.

Parallelization: sectioning and voting

Parallelization has two variants. Sectioning runs independent subtasks concurrently; voting runs the same task several times and collects multiple outputs. Anthropic's sectioning example is a guardrail arrangement in which one model produces an answer while another filters inappropriate requests. Voting examples include having several prompts independently find code vulnerabilities, or judge content suitability at different thresholds. These approaches provide speed or more reliable results through multiple perspectives and attempts.

To calculate the benefit of parallelization, examine the critical path. Total time is determined by the longest path plus the time to combine results. Even ten simultaneous workers cannot make the total shorter than the slowest worker plus the synthesis stage. External API limits can also constrain concurrency. Decide in advance whether to use partial results or rerun everything when one worker fails.

Voting increases cost with the number of calls. Restrict it to ambiguous judgments where a single error is costly.

Orchestrator-workers

In orchestrator-workers, a central LLM dynamically divides work, delegates it to worker LLMs and synthesizes the results. It resembles parallelization, but with a decisive difference: in parallelization, subtasks are known at design time; in orchestrator-workers, they cannot be predicted before seeing the input. Anthropic's examples include coding tools that must edit several files and search tasks exploring multiple sources.

Success depends on the delegation contract. Anthropic's research system article says each subagent needs an objective, an output format, guidance on tools and sources, and clear task boundaries. It also describes two of three subagents duplicating the same supply-chain research because instructions were vague. Include all four elements in a delegation instruction.

Objective: Research only changes to electric vehicle charging rate policy in Korea from 2024 to 2026.
Output format: A table with year, policy name, changes and source URL columns, at most 20 rows.
Tools and sources: Prioritize government and public agency websites; do not use blog summaries.
Boundaries: Other workers handle overseas policies and charger deployment statistics.

Also enforce limits in code on redelegation depth—whether workers may create more workers—and the total number of workers.

Evaluator-optimizer

Evaluator-optimizer is a loop in which one call generates and another evaluates and supplies feedback. Anthropic recommends it when evaluation criteria are clear and the value of iterative improvement can be measured. Check two conditions: does human feedback actually improve the result, and can the model generate such feedback itself? Examples include literary translation requiring subtle criticism like that of a translator, and repeated searches in which an evaluator decides whether further searching is needed.

The loop must end: the evaluator can ask for “a little more” forever. Put a maximum iteration count, a stop when scores cease rising, and a minimum improvement between iterations into code. If the loop ends without convergence, preserve the latest result and evaluation history together for human judgment.

Autonomous agents and stopping conditions

Use autonomous agents for open-ended problems whose required steps are hard to predict or encode in advance. Anthropic recommends sufficient trust in the model's judgment, extensive testing in a sandbox and appropriate guardrails, because costs are higher and errors can accumulate. Add stopping conditions such as maximum iterations to retain control. Chapter 5 covers their detailed design.

The same article names three principles for agent design: keep the design simple, make the agent's planning steps explicit, and refine the agent-computer interface (ACI) through thorough tool documentation and testing. The third principle is the subject of Chapter 2.

The cost of multi-agent systems

In Anthropic's research system example, agents used roughly four times as many tokens as chat, and multi-agent systems roughly fifteen times as many. Analysis of the BrowseComp evaluation found that three factors explained 95% of performance variance, with token usage alone explaining about 80%. The other two were tool call count and model choice. Much of the benefit comes from being able to spend more tokens across multiple windows.

The article therefore concludes that multi-agent systems suit valuable tasks with substantial parallelization opportunities and information that cannot fit into one window. They do not suit tasks where all agents need the same context or have strong dependencies on one another. It also notes that most coding tasks offer less truly parallel work than research, making them less suitable.

Tokens are not the only cost.

Compare cost per successful task, rather than total cost. Doubling cost can be worthwhile if the success rate triples; even unchanged cost is a loss if the success rate stays the same.

Orchestration: should the LLM or code decide?

The OpenAI Agents SDK documentation distinguishes two approaches. LLM orchestration lets the model plan and choose next steps, fitting open-ended work. Code orchestration defines the flow in code, making speed, cost and performance more deterministic and predictable. Documented code orchestration techniques include classifying tasks through structured output to choose the next agent, chaining one agent's output into another's input, running a while loop until an evaluation agent declares a pass, and running independent agents concurrently with primitives such as asyncio.gather.

There are also two ways to connect agents.

A handoff occurs within a single run (turn). The application chooses which agent starts the next turn when it invokes the run. OpenAI's guide calls the same distinction the manager pattern (agents called as tools) and the decentralized pattern (handoffs between peers). Viewed as a graph, manager-pattern edges are tool calls, while decentralized-pattern edges are handoffs that transfer execution. Its general recommendation is to maximize a single agent's capabilities first. There are two signals to split: too many if-then-else branches make the prompt template hard to extend, or similar and overlapping tools cause incorrect selection, rather than simply having many tools. The guide notes that some implementations handle more than 15 distinct tools, while others struggle with fewer than 10 overlapping ones.

Google ADK's sequential, loop and parallel workflow agents are also examples of code orchestration. They follow predefined logic and do not use a model for orchestration, so they are deterministic. However, starting with ADK 2.0 for Python and Go, these template workflows have been replaced by graph-based, dynamic workflows. Check the documentation for your language and version.

Choosing the simplest structure first

Choosing a structure is like climbing a ladder one step at a time. Move up only when you measure failure on the lower step.

Record the decision in an architecture decision record (ADR), a short document explaining what you chose and why. Include the alternatives compared—at least the structure one step below—quality, latency and cost measured on the same task set, rejected alternatives and reasons, and conditions for revisiting the decision. Six months later, it answers “Why is this so complicated?” It also supplies evidence for deciding that a simpler structure is now sufficient.

Going deeper

Code provides the structure; the model provides judgment

The most common robust practical structure combines workflows and agents. Code controls state transitions, business rules, permissions and budgets; the model handles only judgments difficult to express as rules, such as interpretation, planning and text generation. As an analogy, code lays the railway tracks, and the model decides what to tell passengers at each station.

The central rule is separating proposals from execution. The model proposes calling the refund tool next with particular arguments. Code checks whether the proposal is allowed in the current state, fits the budget and needs approval before executing it. These checks limit damage even if the model makes a mistake or is deceived. Encoding every branch, however, requires code changes whenever business rules change. Allow wide latitude for model judgment, but enforce validation at execution boundaries.

Agreement is not verification

In multi-agent or voting arrangements, “all three agents gave the same answer” is weaker evidence than it appears. Agents using the same model, prompt family and search results make the same mistakes. These are correlated errors. Three people agreeing after reading the same false report do not make it true.

Verification gains strength from independence. Separate evidence, rather than just roles. Use different sources, check with a method different from generation—such as code checks or comparison with original text—or assign one agent to find only counterevidence. Before using voting results, measure on an evaluation set whether voting actually reduces errors compared with a single call.

Using frameworks

Agent frameworks make starting easier. Anthropic warns, however, that their abstraction layers can hide underlying prompts and responses, make debugging harder, and encourage complexity when a simple arrangement would suffice. It recommends starting directly with LLM APIs and understanding the underlying code if you use a framework.

There is one practical standard: you must always be able to inspect the exact prompts, tool definitions and tool results actually sent to the model. Establish that visibility first through tracing (Chapter 4) and request logs.

At work

Warm Path: avoiding taxis for structured requests

The travel platform Warm Path built itinerary recommendations around one autonomous agent. It moved between flight, accommodation and attraction tools, averaging 23 tool calls and 40 seconds per response. Even identical requests followed different paths.

The team classified 2,000 request logs. For 87%, city, duration and budget were specified, and the processing stages were almost identical: flight and accommodation lookup → itinerary draft → budget check. Only 13% were open-ended, such as “somewhere quiet to visit with my parents.”

There were three alternatives. First, switch to a stronger model. This was rejected because it left route instability and cost unresolved. Second, use fixed chaining for every request. A 20-case pilot showed lower quality for open-ended requests, so this was rejected too. Third, use one classification call to route requests: structured ones went to fixed chaining with parallel flight and accommodation lookups and a code budget gate; only open-ended ones went to an agent with a maximum turn count. The team chose the third. Low-confidence classifications went to the agent to reduce the risk of lost quality.

Structured requests now took 9 seconds, and cost per request fell by more than 60%. Quality for open-ended requests stayed unchanged on the evaluation set. The ADR included a condition: revisit the structure if the open-ended share exceeds 30%. The evidence did not show that agents were bad. The team had been applying a structure intended for an unpredictable minority to a predictable majority.

Pine Breeze Energy: removing the agent relay

The customer quality team at the city gas company Pine Breeze Energy produces a weekly report on complaint causes. Its original structure had four agents passing work sequentially like a relay: collection, analysis, writing and review. All four reread the same roughly 4,000 weekly complaints. One report used seven times the tokens of the single-agent prototype. Quality scores were slightly higher, but transcripts revealed that the review agent approved the writing agent's conclusions almost unchanged.

The team compared three structures on the same 30 weeks of tasks: the existing relay, a single agent and a hybrid. In the hybrid, code and a small-model router handled collection and classification. Only cause analysis was divided among regional workers for parallel investigation. The drafting phase used an evaluator-optimizer loop with a clear rubric and at most three iterations.

The hybrid cost one third as much per successful report as the relay, with equal or better quality. The review agent was removed because agreement from the same model reading the same data did not provide verification. Code instead checked report figures against source data. The rejected alternative was keeping the same number of agents but using smaller models. That would reduce costs while leaving the structural causes—duplicate reading and coordination overhead—in place.

Common mistakes

1. “Agents are always better than workflows.” An impressive demo can make the structure appear smarter. But autonomy brings cost, latency and unpredictability. Fixed-stage tasks are faster, cheaper and more reproducible with workflows.
2. “More specialist agents improve quality.” Dividing labor may seem to create expertise as in a human organization, but much of the performance gain comes from spending more tokens. For tasks with substantial shared context and dependencies, coordination costs exceed the benefit.
3. “A handoff returns the result to the original agent.” It resembles a function call, but transfers control. Use agent-as-tool when you need to receive the result and continue.
4. “Parallel execution divides total time by the number of workers.” That estimate is tempting, but total time is determined by the critical path and synthesis. Map dependencies and external limits first.
5. “If several agents agree, the answer is correct.” Majority voting looks reliable, but the same models using the same evidence make the same mistakes. Verify through independent evidence or a different method.
6. “Run an evaluator-optimizer loop until quality improves.” Repetition seems likely to help, but without an end condition costs can grow without limit. Put maximum iterations and minimum improvement in code.
7. “The agent is confused by too many tools, so split it.” Tool count is easy to notice, but semantic overlap is often the real cause. Improve names, parameters and descriptions first; split only if that is insufficient.

Summary at a glance

Check the official docs

• Anthropic: Building effective agents — workflow and agent definitions, five patterns and their conditions, three principles, framework advice
• Anthropic: How we built our multi-agent research system — token multiples and performance factors, delegation elements, scaling effort
• OpenAI: A practical guide to building agents — task criteria, single-agent-first approach, manager and decentralized patterns
• OpenAI Agents SDK: Agent orchestration — LLM and code orchestration, handoffs and agents as tools
• Google ADK: Workflow agents — deterministic workflow agents and ADK 2.0 changes

Self-check

1-1. State the distinction between a workflow and an agent in one sentence, and give one example of both approaches in a single system.

1-2. What does each variant of parallelization aim to achieve, and why should voting not be the default at every stage?

1-3. Name the four elements an orchestrator should include when delegating to a subagent, and describe one symptom of a missing element.

1-4. Name two mechanisms that code outside an evaluator-optimizer loop should provide to prevent unlimited spending.

1-5. How do handoff and agent-as-tool differ in conversation control and who produces the final answer?

1-6. Given the observation that token usage explained most performance variance in Anthropic's BrowseComp analysis, what should you compare instead of total cost when deciding whether to adopt a multi-agent system?

1-7. Design a feature that reads 50 internal announcements each week and sends summaries by department. State the simplest structure, a measured signal that it is no longer sufficient, and the next structure to adopt.

Hands-on tasks