Chapter 2. Structured output and tool calling

Learning objectives

• Explain the guarantees provided by instructions, JSON mode and schema-based structured output, and choose the appropriate method.
• Derive schemas from business concepts and adapt them to vendor-specific strict rules.
• Design code to validate results in three stages: syntax, schema and semantics.
• Define tools, implement a tool-calling loop as a state machine with budgets, and return results to the conversation in the correct format.
• Combine parallel calls, tool-choice options and execution permissions to design a safe tool loop.

Introduction

A tax app built a feature to extract expense items from receipt photos and enter them in a ledger. In demos, “Respond in JSON” was enough. Three incidents occurred in the first production week. One response began with “Here are the results”, breaking parsing. Another returned an amount as the string "KRW 12,000", stopping total calculations. The last had perfect format but a total that differed from the sum of item amounts. The first came from failing to enforce format, the second from a schema poorly suited to the business, and the third from assuming that correct format meant correct content.

This chapter covers two features that make model answers usable by programs. Structured output fixes the response shape; tool calling lets the model request application functions. The first part covers schema design and validation. The second proceeds through tool definitions, call loops, returning results, parallel calls and execution permissions.

Key concepts

Structured output is a mold

Dough shaped by hand varies slightly each time, whereas a mold fixes its shape. If “Respond in JSON” is shaping by hand, structured output is a mold. The provider compiles JSON Schema into a grammar and constrains generation to follow it. This is commonly called constrained decoding.

OpenAI explicitly states that JSON mode (json_object) guarantees valid JSON only, not schema compliance. It also warns that a message must contain the word “JSON” somewhere when using JSON mode; without an instruction to produce JSON, the model may emit whitespace until it reaches the limit.

How to enable it for each vendor and choose a method

Choose based on what the result is for. OpenAI recommends function calling when connecting a model to system tools, functions or data, and text.format when shaping the response returned to the user. Gemini similarly distinguishes structured output for final-response format from function calling for actions during a conversation.

The first request using a schema has extra latency while the grammar is built (Anthropic caches compiled grammars for 24 hours after last use). OpenAI and Anthropic SDKs can generate schemas and parse responses from Python Pydantic models or TypeScript Zod schemas, reducing mismatches between handwritten schemas and code types.

Strict schema rules and vendor differences

Strict mode accepts only a subset of JSON Schema. The supported subset differs by provider.

In OpenAI strict mode, represent an optional value by allowing null, rather than omitting the field.

{
  "type": "object",
  "properties": {
    "order_id": { "type": "string" },
    "memo": { "type": ["string", "null"] }
  },
  "required": ["order_id", "memo"],
  "additionalProperties": false
}

Unsupported features in Anthropic produce a 400 error. Anthropic SDK helpers remove unsupported constraints from schemas and move them into field descriptions. Pydantic-based messages.parse() and TypeScript zodOutputFormat() then revalidate responses against the original constraints. When using jsonSchemaOutputFormat() without Zod, or making direct calls without an SDK, the application must perform that validation. Gemini also accepts only a subset, and very large or deeply nested schemas may be rejected.

Derive schemas from business concepts

A schema is a contract between the application and the model, not just a response shape. Decide four things during design.

1. Semantics of missing data. Missing fields, null, empty strings and empty arrays must have different meanings. If a total not printed on a receipt and a total too blurred to read both become the same null, downstream processing cannot choose a path. Include an enum field explaining the reason.
2. Units and representation. Define currency amounts as integers in won (KRW), dates as ISO 8601 (2026-10-20) and ratios as numbers from 0 to 1. If the original wording is needed, use a separate field such as raw_date.
3. What the model generates versus what the server supplies. Do not ask the model to generate values the system knows, such as order numbers, creation timestamps or user IDs.
4. Version. Changing production fields affects screens, storage and batch jobs. Beyond deletions and changes in meaning, even additions break compatibility for consumers validating with additionalProperties: false. Include schema_version, branch consumer handling by version, and update them together.

{
  "type": "object",
  "properties": {
    "schema_version": { "type": "string", "enum": ["2"] },
    "total_krw": { "type": ["integer", "null"] },
    "total_status": { "type": "string", "enum": ["read", "not_printed", "illegible"] },
    "purchased_on": { "type": ["string", "null"], "format": "date" },
    "category": { "type": "string", "enum": ["meal", "supplies", "transport", "other"] }
  },
  "required": ["schema_version", "total_krw", "total_status", "purchased_on", "category"],
  "additionalProperties": false
}

Validate in three stages

Structured output does not end validation. Dividing validation into three stages clarifies failure causes and responses.

Stage 0 is easily missed. Anthropic states that refusal (refusal) and output-limit (max_tokens) responses may not match the schema; OpenAI likewise says to check status: "incomplete" first. Stage 3 includes relational checks such as “sum of item amounts = total”, “refund amount ≤ payment amount” and “end date ≥ start date”. As Gemini puts it, even syntactically correct JSON always requires application-side value validation.

There are subtler traps. Anthropic says structured output does not guarantee the case of enum values, recommending case-insensitive comparison and avoiding enum values that differ only by case. Compiled schemas are also cached separately from message content for up to 24 hours after last use, so sensitive information such as customer names should not appear in field names, descriptions or enum values.

A tool definition is an execution contract

Tool calling (function calling) tells the model which functions exist and lets it propose names and arguments when needed. The model does not directly execute application-defined tools; the application does (except server tools such as web search executed by the provider). A tool definition consists of a name, description and input schema. The description is the model's entire instruction manual.

Anthropic identifies detailed descriptions as the largest influence on tool performance. It recommends at least three or four sentences per tool covering what it does, when to use or avoid it, each parameter's meaning and limitations such as information it does not return.

Other design principles follow.

• Add examples for format-sensitive arguments. Anthropic provides input_examples for examples that pass schema validation.
• Combine related operations into one tool with an action parameter. For multiple services, add prefixes such as billing_ and calendar_.
• Return only information and stable identifiers needed for the next decision; do not ask the model to fill in already-known values such as user IDs (OpenAI).
• Keep tool counts low. OpenAI recommends fewer than 20 functions at the start of a turn; Gemini recommends no more than 10–20 active tools.

Build the tool-calling loop as a state machine

Tool calling runs in a loop rather than ending with a single request.

1. Send a request with tool definitions.
2. Extract tool calls (name, arguments and call ID) from the response.
3. Validate arguments and permissions, then execute in the application.
4. Add results to the conversation, paired with call IDs, and request again.
5. End on a final answer; if more tool calls arrive, return to step 3.

Starting with a simple while loop can produce endless calls, incorrect termination and runaway costs. Thinking in states makes missing paths visible. The cycle is “generating → call requested → validating → executing → returning results → generating”, ending in “complete” or “budget exceeded”. Set three budgets: step count, elapsed time and token cost.

const budget = { maxSteps: 6, deadlineMs: Date.now() + 30_000, maxTokens: 60_000 };
let usedTokens = 0;
for (let step = 0; ; step++) {
  if (step >= budget.maxSteps || Date.now() > budget.deadlineMs || usedTokens > budget.maxTokens) {
    return handOff("budget_exceeded"); // Connect to a human agent; preserve partial results
  }
  const res = await callModel(messages, tools);
  usedTokens += totalTokens(res.usage); // Include cache tokens
  const outcome = classify(res.stop_reason); // Chapter 1's classifier
  if (outcome !== "needs_tool") return finish(res, outcome);
  const results = await runTools(res); // Validate → execute → result envelope
  messages.push(assistantTurn(res), toolResultTurn(results));
}

You do not have to write the loop yourself. Helpers such as the Anthropic SDK's Tool Runner automate tool execution and result-return loops. Budgets, permission checks and audit records remain the application's responsibility in every case.

Return results to the conversation

Match calls and results by call ID. Formats differ by vendor.

Anthropic has strict placement rules. tool_result must appear in the user message immediately following its tool_use, with no intervening messages, and before text in the content array. Text appended immediately after results can also cause empty responses. OpenAI's arguments is a JSON string requiring parsing; a function without a return value should return a string indicating success or failure.

Failure is also a result. With Anthropic, use is_error: true and explain what went wrong and what to try next. “Inventory system did not respond; retry in 60 seconds or check another branch” is better than “failed”. When invalid arguments receive an error result, Claude usually corrects and retries two or three times. Different result shapes are hard to interpret, so define a shared envelope.

{ "status": "needs_info", "data": null, "error": null, "hint": "The unit and apartment number for the visit address are required" }

Values such as ok, error and needs_info are enough for status. Cap result size and exclude personal information the model does not need.

Parallel tool calls

A model can request several tools in one response. Anthropic explains that the API does not determine execution order: run independent reads concurrently, and operations with side effects, shared state or ordering requirements sequentially. Regardless of execution method, follow these rules.

• Return one result for every call. Even a call you choose not to execute needs an is_error: true result with a brief explanation such as “Not executed because the earlier call failed”.
• With Anthropic, put all results in one user message. The documentation warns that splitting results across messages conditions the model to avoid parallel calls, reducing parallel usage.
• Parallel calls can be disabled. Anthropic's disable_parallel_tool_use: true belongs inside tool_choice, not at the top level; with auto, it allows at most one tool per response. OpenAI's parallel_tool_calls: false allows zero or one.

Gemini distinguishes independent parallel calls from compositional calls, where an earlier result becomes a later call's input. The application must enforce ordering when dependencies exist.

tool_choice and forced calls

tool_choice determines whether and which tools the model uses.

Forced-call support differs by model. Anthropic states that Claude Opus 5.5, Sonnet 5.5, Fable 5.1 and Mythos 5.1 reject any and tool with 400 errors. Use auto with strict tools, or structured output when a fixed JSON response is needed. Even supported models omit thinking and explanatory text before tool calls when calls are forced. In Anthropic, changing tool_choice invalidates the cached messages portion. OpenAI says allowed_tools, which restricts callable tools without changing the tool list, helps preserve cache savings.

Reasoning models and tool loops

Reasoning models return reasoning information alongside tool calls. Send this information back unchanged in the next request.

• OpenAI: return reasoning items received with tool calls alongside function results. Use previous_response_id or include earlier output items unchanged in the input. In stateless use, pass the reasoning item's encrypted_content.
• Anthropic: when returning tool results, send the latest assistant message's thinking and redacted_thinking blocks (thinking partially hidden for safety) unchanged, including signature. Editing, reordering or removing these blocks causes a 400 error; empty blocks are no exception.
• Gemini: when managing history statelessly, resend model-generated steps unchanged because they contain required signatures. The SDK handles this automatically.

Problems often come from middleware that filters blocks or reserializes history to save tokens. The rule is “history is append-only”.

The server holds execution permissions

Strict mode guarantees the shape of arguments only. Whether those arguments may be executed is a separate question. The OWASP Top 10 for LLM Applications 2025 calls the risk of harmful actions caused by excessive functionality, permissions or autonomy LLM06 Excessive Agency, and passing model output to other systems without sufficient validation or sanitization LLM05 Improper Output Handling.

1. The server supplies user and organization identifiers from the session. Do not let the model select customer IDs as arguments.
2. Recheck permissions and business rules (state, limits and ownership) immediately before execution. Gemini also recommends validating function calls before executing them.
3. Separate read tools from write tools, allowing writes only within the minimum necessary scope.
4. Require human approval for hard-to-reverse operations. Save an approval-pending state and return the decision as a tool result to continue the conversation.
5. Do not trust external text inside tool results (Chapter 1, trust boundaries).

Going deeper

When the number of tools grows

As systems are connected, tools grow into dozens or hundreds. Both cost and accuracy suffer. Tool definitions are billed as input tokens (explicitly stated by both OpenAI and Anthropic), and Anthropic also adds a tool-use system prompt. Many similar tools increase selection errors.

Anthropic offers a tool-search tool that finds and loads tools as needed from thousands; OpenAI supports deferred loading and tool search on recent models. Anthropic invalidates the entire cache when definitions change, so rebuilding the list at every stage can offset savings. Measure tool-selection accuracy on an evaluation set whichever strategy you use.

Structured results during streaming

Structured output and tool arguments can also be streamed. You can display them sooner, but execution must wait for completion.

• Anthropic sends tool input as partial JSON strings in input_json_delta. Accumulate fragments and parse at content_block_stop. For smaller chunks of large parameter values, fine-grained streaming can be enabled per tool with eager_input_streaming.
• OpenAI sends fragments in response.function_call_arguments.delta and complete arguments in .done.
• Gemini describes streamed fragments as partial JSON strings that concatenate into the final object.

The rule is simple: use partial results for previews, but execute tools or save results only after complete arguments pass all three validation stages. If a stream breaks mid-way, Anthropic says tool-use blocks cannot be partially recovered. Request again rather than joining received fragments and executing them (Chapter 4).

At work

Sandgrain Accounting: redesigning a receipt-extraction schema

Sandgrain Accounting, a bookkeeping app for small business owners, extracts expense items from OCR text of receipt photos. After dozens of daily parsing failures and type errors with instruction-based JSON, developer Seong redesigned the schema.

The first draft made the total optional and omitted it when unreadable. But OpenAI strict mode required every field in required, and the app needed to distinguish “not printed” from “too blurred to read”: the former could use the sum of items, while the latter required asking the user for another photo. So total_krw became a nullable integer and total_status an enum. Amounts were fixed as integer won, dates as ISO format, and original date text was captured separately in raw_date. The server added the ledger-item ID and registration time.

Testing the same feature with Claude revealed another issue: the minimum: 0 constraint requiring nonnegative amounts was unsupported in Anthropic structured output. The team moved it into the field description and rechecked it in application code after receiving a response. Semantic validation sent users to a confirmation screen if the difference between item sums and the total exceeded KRW 10.

Adding more examples to JSON mode could not prevent schema mismatches. Returning zero for unreadable amounts was rejected because it was indistinguishable from an actual zero-won receipt. A month after deployment, parsing failures disappeared; 1.2% of receipts went to confirmation for mismatched totals and were corrected by users.

Fingertip Repair Center: setting boundaries for a repair-intake agent

Appliance repair company Fingertip Repair Center is building a chat-based intake agent. It has three tools: warranty lookup (lookup_warranty), available visit slots (find_slots) and ticket creation (create_repair_ticket). Three incidents occurred in the pilot. A user supplied someone else's product serial number and received its warranty details. The model requested slot lookup and ticket creation together, creating a ticket before availability was checked. One vague request also ran the loop nine times, consuming budget.

Lead engineer Do changed the structure before changing prompts.

1. Customer ID was removed from tool arguments. The server supplies it from the login session and checks before execution that the serial number belongs to a product registered to that customer.
2. create_repair_ticket now requires a slot_hold_id returned by find_slots. Holds expire; a missing or expired hold prevents execution and returns needs_info asking for another lookup. Even if parallel requests arrive, server-enforced dependencies preserve ordering.
3. The two read tools may run concurrently. If a write appears in the same response, reads run first and the write is then validated. Unexecuted calls receive error results explaining why.
4. The loop budget is six steps and 30 seconds. Exceeding it hands the conversation to an agent with a summary.

An instruction alone saying “Do not look up other people's products” could not prevent violations if the model ignored it, so it was used together with server checks. Forcing ticket creation was rejected because the intended model did not support it and because it removed room to ask for missing information. Intake errors disappeared, while the human-handoff rate stayed at 7%.

Common mistakes

• JSON mode also enforces the schema. It guarantees syntax only. Schema-based structured output guarantees fields and types.
• Passing the schema means correct content. Results can have correct format and wrong values. Check relationships involving totals, dates and states through semantic validation.
• Optional fields can simply be left out of required. In OpenAI strict mode, every field must be in required, using nullable types for optional values.
• The provider enforces every constraint in the schema. Support differs. Anthropic does not accept numeric or string-length constraints; validate them in the application.
• Unexecuted calls do not need results. Every call must have a result. Report nonexecution as an error result.
• Strict mode makes execution safe. Strict mode guarantees shape only. The server checks permissions, ownership and business rules immediately before execution.
• Parallel results can be split across messages. Anthropic requires a single user message; splitting results reduces parallel calls.

Summary at a glance

Check the official docs

• OpenAI — Structured outputs: read differences from JSON mode, supported schemas and limits.
• OpenAI — Function calling: read the call flow, strict rules and tool_choice options.
• Anthropic — Structured outputs: check supported and unsupported schema features and complexity limits.
• Anthropic — Define tools: read description best practices and the forced-call support table.
• Anthropic — Handle tool calls: read tool_result placement rules and writing is_error results.
• Google — Gemini API function calling: read calling modes and best practices.

Self-check

2-1. List four decisions when deriving a schema from business concepts. Show how to distinguish “cannot read the value” from “the value is absent”.

2-2. Give two differences between OpenAI and Anthropic strict-schema rules, and describe precautions for managing schemas in a service using both providers.

2-3. Define syntax, schema and semantic validation in one sentence each, and state what must be checked before these three stages.

2-4. Define fields for a common tool-result envelope and explain why failure results should state what to try next.

2-5. List the states needed for a tool-loop state machine, the three loop budgets and what happens when a budget is exceeded.

2-6. Give three examples of values the server should supply rather than letting the model select them as arguments. Explain why in relation to risks in the OWASP Top 10 for LLM Applications.

2-7. Design an internal meeting-room booking assistant. Create a table of tools, essential description contents, server-supplied values and checks immediately before execution.

Hands-on tasks