Chapter 3. Performance and cost optimization

Learning objectives

• Break a request's latency into stages and choose the improvement with the greatest effect.
• Handle streaming events correctly and distinguish what streaming reduces from what it does not.
• Arrange prompts to improve cache hits and calculate cache break-even points.
• Choose tasks suited to batch APIs and service tiers, and retrieve results safely.
• Design model-tier routing and escalation conditions, judging latency/cost trade-offs by cost per business task.

Introduction

One team responded to complaints about slow answers by switching to the most expensive model. Responses became slower and bills tripled. Another halved its prompt to cut costs, reducing quality while barely changing latency. Both acted before measuring. Latency and cost are sums of several stages and components; only measurement tells you which dominates.

This chapter starts with measurement. It first covers breaking latency into stages and OpenAI's latency-reduction principles, followed by streaming to reduce perceived delay, prompt caching to reuse repeated input, batch APIs and service tiers for cheaper nonurgent work, and routing to choose a suitable model per request. Finally, it compares these approaches using one measure: cost per business task.

Key concepts

Break latency into stages

User waiting time is the sum of several stages: application queuing, input preprocessing, model processing, tool execution, postprocessing and network delivery. Model processing itself has two parts: reading all input and generating output tokens one at a time. Measure three separate metrics.

Also consider when the user sees the first usable result. If the first fragment arrives quickly but the screen needs a complete structured result, perceived delay is close to total response time. Complaints arise in the tail, so watch p95 (the boundary of the slowest 5%) rather than the average.

Seven principles for reducing latency

OpenAI's latency-optimization guide lists seven principles. They apply across providers.

The key is that output has the greatest influence on latency. Introductions repeating the question, long conclusions and unnecessary full rewrites all increase latency and cost.

How streaming works

Streaming receives a response bit by bit as server-sent events (SSE). A dispatch notification does not change a parcel's arrival time; similarly, streaming shows initial content sooner but does not reduce generated tokens or cost.

Anthropic streams events in a defined order: message_start → content_block_start, repeated content_block_delta, content_block_stop for each block → message_delta → message_stop, with possible ping events between them. Watch these details.

• stop_reason is null in message_start and finalized in message_delta.
• usage in message_delta is cumulative. Summing usage values across every delta event overcounts usage.
• An error event, such as overloaded_error, can arrive after HTTP 200. Handling errors only through status codes misses it.
• If you do not need raw events, use SDK accumulation helpers—Python get_final_message() or TypeScript finalMessage()—to get a complete message.

The OpenAI Responses API sends semantic events such as response.created, response.output_text.delta, response.completed and error. Gemini streams through SDK options or REST alt=sse.

Design the entire streaming path

Users may still see a blank screen with API streaming enabled. If an intermediate layer buffers responses and sends them at once, streaming loses its benefit. Check the whole path.

1. Disable response buffering on SSE routes in reverse proxies and API gateways. Compression settings may also accumulate chunks.
2. Forward received chunks immediately, but group screen updates into reasonable units (sentences or 100 ms) to reduce browser work.
3. When users close the window or cancel, propagate cancellation upstream through the HTTP client's cancellation signal, such as AbortController. Finishing output no one sees only costs money.
4. Determine whether postprocessing can work incrementally. OpenAI warns that streaming makes output moderation harder. Determine first whether content can be displayed only after moderating the entire response.
5. Accumulate tool-argument or JSON fragments, but execute and save only after completion and validation (Chapter 2).

How prompt caching works

Instead of rereading the same meeting documents from the beginning every time, imagine marking what you have read and reading only newly added pages. Prompt caching stores processed request prefixes for reuse in later requests. If even one character differs in the prefix, reuse stops at that point.

Caching does not store and return the output. A new response is generated each time; OpenAI states that caching does not change output-token generation. This fundamentally differs from response caching, which reuses an earlier answer unchanged (Going deeper). Prompt caching provides two benefits: cached input is billed at a lower rate, and reduced input-processing time brings the first token sooner.

Comparing caching across vendors

With Anthropic caching, input_tokens means only tokens after the last breakpoint. Add all three fields for total input.

Total input tokens = cache_read_input_tokens + cache_creation_input_tokens + input_tokens

Gemini supports only implicit caching in the Interactions API and on Gemini 4 Argon and later models. Explicit caching, where you create and manage a cache yourself, is available only through generateContent on supported earlier models.

Arrange prompts for more cache hits

All three providers share one rule: stable content first, changing content last. A single current-time line at the beginning can cause thousands of following tokens to miss the cache.

1. Fix the order. Anthropic builds the prefix as tools → system → messages. Put tool definitions, fixed instructions, reference material and examples first; time, user information and questions last.
2. Put breakpoints at the end of stable content. A breakpoint in a changing block causes repeated writes.
3. Stabilize serialization. Different tool ordering, JSON key order or whitespace makes a different prefix even if meaning is identical. Building tool lists from sets or hash maps can change their order.
4. Know what invalidates caches. In Anthropic, changing tool definitions invalidates the whole cache; changing tool_choice or adding/removing images invalidates the messages portion. Changes to top-level effort or output_config.format also invalidate the cache.
5. Check minimum length. A prefix below the model's minimum silently skips caching even when marked.
6. Measure. Compare cache-read ratios before and after deployment; use tools such as Anthropic's cache diagnostics to find where consecutive request prefixes diverge.

OpenAI's allowed_tools preserves caching by leaving the tool list unchanged while restricting what may be called. prompt_cache_key affects routing only on earlier models; from GPT-5.6 onward, it mainly accounts for cache usage per customer. GPT-5.6 and later models place a breakpoint at the end of recent messages by default. If that ending changes each time, add prompt_cache_breakpoint at the end of stable content to get hits. Anthropic caches are not shared across organizations and are also isolated by workspace in the Claude API.

Cache break-even points

Cache writes cost more than ordinary input, so enough reuse is needed. Set prefix-processing cost to 1 and calculate with Anthropic's multipliers.

At a 0.1× read rate, the five-minute cache saves from the second request within its lifetime, and the one-hour cache from the third (models with lower read multipliers save more). If requests are spaced beyond the lifetime, every call is a write: five-minute caching costs 25% more and one-hour caching 100% more than no cache. Choose a longer lifetime or disable caching in that case. Lifetime starts when the request starts, so long generation leaves less time for reuse. There is also a throughput benefit: cache reads do not count toward ITPM for most Anthropic models, allowing more input under the same limit when hit rates are high.

Batch APIs

Use cheap ordinary post for parcels that do not need same-day delivery. Batch APIs collect large volumes of requests that need no immediate response, process them asynchronously and discount the price.

OpenAI and Anthropic state that result order may differ from input order, so always match by identifier. Gemini's API reference specifies input order, but keys are safer for code that must also work with other providers. Anthropic validates requests asynchronously, so format errors may appear only after batch completion. It recommends first testing one request of the same format through the Messages API. Failed-request billing also differs: Anthropic does not charge errored, canceled or expired requests; OpenAI cancels unfinished requests on expiry and bills completed requests only. Batching can be combined with caching, but Anthropic calls within-batch cache hits best-effort and recommends a one-hour cache for shared context. Gemini's batch documentation says context-caching rates apply to cache hits, so check actual model-specific pricing. Batch limits are managed separately from synchronous API limits.

Choose a service tier

The same model can process requests at different service tiers.

Tier selection depends on the value of time. Send evaluations or data enrichment that can wait minutes to a cheap tier; use fast tiers only for customer-facing parts missing latency targets. If using a fast tier, log the actual tier in responses and monitor downgrade rates.

Model-tier routing and escalation

Sending every request to the largest model is like using a heavy truck for a neighborhood errand. Routing sends requests to different models according to their nature; escalation reruns work on a larger model when a small model's result falls below criteria.

Do not base escalation on a model's self-reported confidence: confidence and correctness do not always align. Use verifiable signals such as schema compliance, required facts and absence of prohibited expressions. Evaluate quality, latency and cost separately for each route. Route reasoning effort the same way for reasoning models.

Judge latency/cost trade-offs

Each approach gains and gives up different things.

Compare cost per business task, rather than cost per call.

Cost per business task = (all call costs + retry/escalation costs + human review costs) ÷ tasks passing quality criteria

Even a model costing half per call loses money if retries and human corrections triple. Set deadlines and failure costs first, then choose the lowest-cost-per-task combination within them.

Going deeper

When is response caching dangerous?

Response caching returns an earlier answer unchanged. With many identical questions, it can virtually eliminate latency and cost. Semantic caching also returns the same answer for differently worded but similar questions, using embedding similarity to find earlier questions. Its power brings clear risks.

1. Personalization and permissions. If answers contain user-specific information, they may reach someone else. Include organization, permissions, language and source version in cache keys, or avoid caching personalized answers.
2. Freshness. When source documents change, cached answers become wrong. Include document versions in keys and invalidate affected entries on revision.
3. Similarity threshold. A low threshold can give identical answers to similar but distinct questions, such as refund windows and exchange windows. Set thresholds with evaluation sets of false cache hits.
4. Modification operations. Do not cache requests requiring tool execution.

Prompt caching does not have these risks because it generates new output every time. The two caches operate at different layers. Use response caching for selected questions with fixed answers, such as established policy information, and prompt caching across the rest of the conversation.

Compare optimization experiments fairly

Optimization effects are often misread in four ways. First, mixing cache states: the first request is slower and more expensive because the cache is empty. Measure cold and warm caches separately. Second, ignoring quality: shorter output and smaller models can reduce latency and cost while lowering quality. Fix the same evaluation set and pass criteria. Third, ignoring load: concurrency changes latency distributions, so measure p95 near actual peak concurrency. Fourth, making several changes at once: switching models, prompts and cache placement together hides the source of improvement. Change one thing per measurement and record cost per task and p95.

At work

Mulmaru Language Academy: explaining a 14-second blank screen

Online language school Mulmaru Language Academy corrects learners' English compositions. Results take 14 seconds on average after submission, with a blank screen throughout. Operations proposed a top-tier model, but developer Tak first measured the stages.

API streaming was already enabled. But the internal API gateway relayed SSE with response buffering on, so the browser received nothing until completion. Output generation also dominated latency: the model rewrote the entire composition, then added long explanations.

Tak made two changes: disabling buffering on the gateway's SSE route and changing the response format to a list in which each entry contains the sentence needing correction, the corrected sentence and a one-line explanation. Full rewrites became a separate request only when learners wanted them. Average output fell from 950 to 340 tokens. The first correction appeared after 2.1 seconds, completion took 5.6 seconds, and cost fell 60%. Correction accuracy on the evaluation set was unchanged.

Other proposals were rejected. The top-tier model generated output more slowly. Reducing prompt examples was not attempted because input was only about 2,000 tokens and little improvement was expected.

Deullyeok Distribution: splitting review replies into three paths

Online produce retailer Deullyeok Distribution drafts replies to about 9,000 product reviews daily. Costs rose because all reviews used a synchronous API and top-tier model. Company policy allows 48 hours for ordinary reviews and 30 minutes for complaints rated one or two stars.

Platform engineer Wi divided routes by deadline.

1. Complaint reviews (about 11%) use a synchronous path. A small model drafts replies; a validator checks required elements (apology and action guidance) and prohibited elements (promised compensation amounts and unverified causes). Failures escalate to a mid-tier model. The first week's escalation rate was 14%.
2. Ordinary reviews go to batch APIs twice daily. Review IDs become custom_id values for result matching; only expired or errored items enter the next batch.
3. Reply guidelines and tone examples (about 3,000 tokens) stay fixed at the prompt's start; review text and product information go last.

The team compared cost per task, including review time for drafts needing human correction. Using a small model for everything had the lowest call cost, but complaint corrections increased, making cost per task higher. The final setup reduced cost per task to 31% of the original. They also considered reusing replies through response caching, but rejected it because replies vary by order history and product condition.

Common mistakes

• Streaming reduces cost. It reduces perceived delay; generated tokens and cost remain the same.
• Halving the prompt halves latency. Output length dominates latency. Reducing input usually has little effect.
• Longer cache lifetime raises the hit rate. If prefixes differ every time, lifetime does not help. Fix ordering and serialization first.
• Adding cache_control guarantees caching. Prefixes below model-specific minimum lengths silently skip caching. Check usage fields.
• Batch results arrive in input order within a few hours. OpenAI and Anthropic do not guarantee order, and timing is defined only within each provider's stated window, such as up to 24 hours. Match by identifier and prepare for expiry.
• Cache reads never count toward rate limits. Most Anthropic models exclude them from ITPM, but OpenAI includes cached input in TPM.
• Starting with a small model is always cheaper. Judge cost per task including escalation, retries and human corrections.

Summary at a glance

• Measure TTFT, output speed and total time separately, using p95. Output length is the largest variable.
• Streaming reduces perceived delay. Check the whole path, including gateway buffering, cancellation propagation and moderation timing.
• Anthropic streams: stop_reason arrives in message_delta; usage is cumulative; error events can follow 200.
• Caching reuses prefixes. Stable content first, variable content last, stable serialization, minimum-length checks.
• At a 0.1× read rate, Anthropic's five-minute cache saves from the second use, and one-hour cache from the third. Request intervals longer than the cache lifetime make caching more expensive than uncached requests.
• All three providers discount batches by 50%. OpenAI/Anthropic do not guarantee order. Match by identifier and prepare for expiry.
• Cheap service tiers trade away speed; fast tiers require premiums and downgrade management.
• Escalate using verifiable signals. Compare cost per business task.

Check the official docs

• OpenAI — Latency optimization: read the seven principles and examples.
• Anthropic — Streaming messages: read event flow, cumulative usage, error events and recovery.
• Anthropic — Prompt caching: check prefix order, lifetime, price multipliers, minimum lengths and invalidation tables.
• OpenAI — Prompt caching: read default enablement, retention, prompt_cache_key and usage fields.
• Google — Gemini API context caching: check implicit-caching conditions and minimum tokens.
• OpenAI — Batch API, Anthropic — Batch processing, Google — Batch API: compare limits, expiry and result matching.

Self-check

3-1. Define three separate request-latency metrics and explain why p95 matters more than the average.

3-2. Give two reasons users might see a blank screen even with API streaming enabled, and list checks for the streaming path.

3-3. Calculate how many uses of the same prefix within its lifetime make Anthropic's one-hour cache (2× writes, 0.1× reads) worthwhile. Compare with the five-minute cache (1.25× writes).

3-4. State what prompt caches and response caches each reuse. Explain risks and safeguards when response caching answers containing user-specific information.

3-5. Give three signals to replace self-reported model confidence as escalation conditions, and explain how escalation rate affects cost per task.

3-6. Build a decision table choosing processing paths (synchronous, service tier, batch) and model tiers for three tasks with immediate, four-hour and three-day deadlines. Explain each choice.

Hands-on tasks