Chapter 4. Reliability and error handling

Learning objectives

• Read status codes, body error codes and headers to distinguish retryable errors, errors requiring fixes and cases for human judgment.
• Explain rate-limit structure and the different causes of 429 responses, and control traffic.
• Design a single-layer retry policy with jitter and budgets, and define layered timeouts and deadlines.
• Safely repeat operations with side effects using idempotency keys, and recover unknown outcomes.
• Recover interrupted responses and partial failures while preventing failures from spreading to other features.

Introduction

At two in the morning, a manuscript proofreading job stopped. An operator restarted it from the beginning upon arriving at work; by noon, editors reported two copies of each chapter's corrections. Logs showed mixed 529 and 429 errors overnight, with retry logic repeating every failure infinitely at one-second intervals. The response to the incident had caused more harm than the incident itself.

Failure is routine in systems using external APIs. What matters is classifying failures, applying a defined response for each kind, and avoiding duplicate work. This chapter covers error classification, rate limits, retries and backoff, timeouts, idempotency, interrupted responses and partial failures, and failure isolation.

Key concepts

Success, failure and unknown outcomes

Requests have three outcomes, not two: a success response, a failure response or no response at all. The third is an unknown outcome. The server may have completed the operation before the connection broke, or may not have started processing it at all.

An ordinary model call with an unknown outcome can simply be resent. It costs a little more but has no external side effects. Operations with side effects are different. Treating an unknown refund, booking or email outcome as a failure and retrying may execute the operation twice. Handle modification operations with unknown outcomes in this order.

1. Look up the actual state using the same business key.
2. If already applied, treat the operation as successful.
3. If confirmed unapplied, execute again with the same idempotency key.
4. If lookup is impossible, hold the operation and alert an operator instead of automatically retrying.

Error classification table

Provider names differ, but status codes have similar meanings.

Gemini explicitly says not to retry client errors such as 400, 402 and 403. OpenAI likewise explains that invalid requests produce the same result until fixed.

A decision table for retryable errors

Status codes alone are insufficient. Read body error codes and headers and answer three questions.

Retry automatically only if all three answers are yes. If the first is no, fix the request or notify a person. If the second is no, use the unknown-outcome procedure. If the third is no, mark the operation as failed and take an alternative path.

Rate-limit structure

Rate limits cap requests and tokens within a time period. Units differ by provider.

Anthropic uses a token-bucket algorithm. Like a bucket filling continuously at a fixed rate while requests draw water, capacity does not reset all at once at a fixed time. A 60-per-minute limit can therefore be enforced as one per second, and a brief burst can trigger a 429 response even when the minute's total is within limits. Sudden usage growth can hit acceleration limits, so the documentation recommends ramping traffic gradually.

Response headers show remaining capacity. Anthropic provides per-item limits, remaining amounts and reset times through headers such as anthropic-ratelimit-requests-remaining and anthropic-ratelimit-input-tokens-remaining; OpenAI provides x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens. Note that limit calculations also vary across providers. OpenAI calculates token limits using the larger of max_tokens and estimated tokens, including cached input in TPM. Anthropic OTPM counts actual generation only, and most models exclude cache reads from ITPM.

How to read a 429

The same 429 can require opposite responses depending on its cause.

Also remember that manually configured Anthropic organization or workspace spend limits return 400 invalid_request_error, not 429 (Claude Code workspace limits are an exception, returning 429 with retry-after). The Gemini codes in the table refer to the Interactions API; generateContent uses a different format, such as RESOURCE_EXHAUSTED. Branch by cause, not status alone.

Exponential backoff and jitter

For retryable errors, decide how long to wait. Exponential backoff doubles the delay with each attempt. Jitter adds randomness so thousands of clients affected by one incident do not all retry simultaneously. If a queue collapses in front of an elevator and everyone rushes back exactly ten seconds later, the queue jams again.

function backoffMs(attempt: number, retryAfterSec?: number): number | null {
  const base = 500, cap = 30_000;
  if (retryAfterSec !== undefined) {
    const wait = retryAfterSec * 1000 + Math.random() * 1000; // Minimum wait + small random delay
    return wait > cap ? null : wait; // Beyond cap: null means finalize failure without retrying
  }
  const ceiling = Math.min(cap, base * 2 ** attempt);
  return Math.random() * ceiling; // Full jitter: choose between zero and the ceiling
}

OpenAI recommends treating Retry-After as a minimum delay and adding a little randomness, with limits on both attempt count and total retry time. It also warns that failed requests consume per-minute limits, so repeatedly resending the same request only worsens the situation.

Retry in one place, within a budget

Retries hide at multiple layers. Anthropic's official SDK retries connection errors, 429 and 5xx twice by default with exponential backoff, follows retry-after and can be adjusted with max_retries. Gemini's Python SDK retries temporary errors up to four times; OpenAI's official SDK retries connection errors and 408, 409, 429 and 5xx twice by default. Application wrappers, job queues and API gateways can add their own retries, multiplying attempts.

The AWS Builders' Library calls retries “selfish”: they consume more server capacity to improve your request's probability of success. It recommends retrying at one point in the stack, only while dependencies appear healthy. Put this into practice as follows.

1. Choose one retry-owning layer. If the SDK owns retries, disable wrapper retries; if the wrapper owns them, set SDK retries to zero. Setting names differ by SDK and language, for example Python max_retries and TypeScript maxRetries.
2. Set retry budgets: attempt count, total waiting time and retry ratio relative to all requests (for example 10%). A client-side token bucket that refills a little on normal responses and spends tokens on retries prevents retry explosions during incidents.
3. Log the original request ID and attempt number. You need the actual number of deliveries per request to enforce budgets.

Design timeouts in layers

One timeout for all waiting either cuts requests too early or holds them too long. Use three layers.

AWS describes choosing a high percentile, such as p99.9, from downstream latency distributions and accepting a small false-timeout rate. Propagate the deadline downstream. Give tools the remaining time, and do not start work that would finish after the deadline.

Use streaming or batch APIs for long generation tasks. Anthropic recommends them for requests that may exceed ten minutes because some networks terminate long-idle connections. SDKs check that nonstreaming requests are unlikely to exceed ten minutes and configure TCP keep-alive; the OpenAI SDK also defaults to a ten-minute timeout. OpenAI says synchronous responses are canceled by closing the connection, while server-side work such as background mode requires explicit cancellation. If only an intermediate connection closes and the app server's upstream request remains alive, generation and billing continue.

Idempotency: two presses, one action

Pressing an elevator call button repeatedly still calls one elevator. Idempotency means multiple identical requests have the same result as one. The HTTP standard (RFC 9110) defines PUT, DELETE and safe methods as idempotent, and says automatic retries of non-idempotent requests require knowing that the request is effectively idempotent or having a way to confirm the original was not applied. AWS likewise says APIs with side effects are unsafe to retry unless they provide idempotency.

Build application-level idempotency as follows.

1. Assign one key per operation and reuse it for every retry. Examples include a tool-call-ID-based key or a business key such as refund:{order_number}:{refund_round}.
2. The receiver stores a unique key, state (processing, complete, failed), request fingerprint (hash) and result. Claim the key atomically, for example with a unique-constraint insert, so concurrent requests sharing the same key execute only once.
3. Return the saved result when the same key arrives again. Reject the same key with different content as a conflict.
4. Retain keys longer than the possible retry window.

Model-generated tool-call IDs are unique per call and useful key material. Resending the same tool call after a network error carries the same key, allowing duplicate detection. Generating a fresh key on every retry defeats the mechanism. If the model request itself can be repeated and the model can propose the same operation as a new call, also include a business identifier such as order number so the receiver recognizes the same operation. Incoming events need the same protection: OpenAI warns that webhooks may occasionally deliver an event twice and recommends deduplicating with the webhook-id header.

Recover interrupted responses

Streaming can fail after HTTP 200. Handle errors at the event level and define a recovery strategy in advance.

• Anthropic: save received content and ask to continue. For Claude 4.5 and earlier, use the received portion to start a new assistant message; for Claude 4.6 and later, send it with a user message asking to continue from the break. Tool-use and thinking blocks cannot be partially recovered; continue only through the last text block and request the rest anew. Continuations may duplicate or drift, so starting again is safer when format matters, as with structured output.
• OpenAI: run long tasks server-side with background: true and poll status. Combined with streaming, save each event's sequence_number as a cursor and reconnect with starting_after to resume there. In zero data retention (ZDR) projects, background mode runs with store=false, with response data temporarily retained for about ten minutes for polling.

Showing an incomplete result and declaring an operation complete are two different things. Keep previews from interrupted responses in a “generating” state and save only complete, validated results.

Recover partial failures

Jobs with multiple units often fail only in part. The rule is “preserve successes; retry only failures according to cause”.

• Parallel tool calls: return failure results and let the model decide the next action. Unexecuted calls also receive results explaining why (Chapter 2).
• Batches: separate success, error, cancellation and expiry. Resubmit expired, canceled and server-error items; do not resubmit request-format errors until fixed. Do not regenerate successful items.
• Multistage pipelines: save per-item state (pending, running, complete, failed, quarantined) and checkpoint after each stage. Skip completed items on restart.
• Irreversible operations: define compensation procedures, such as correction emails or manual handling, for irreversible actions such as sending emails.

Move repeatedly failing inputs to a dead letter queue so they do not block other work, and escalate them for human investigation.

Prevent failure from spreading

When an external API remains slow or fails, waiting threads and connections accumulate and slow unrelated features. Like bulkheads on a ship, controls must contain the damage.

A circuit breaker has three states. As in an electrical circuit, current flows when closed. In the closed (normal) state, calls pass through; in the open (blocked) state, calls are immediately routed to the fallback; in the half-open (testing) state, some successful probes return the circuit breaker to the closed state. Waiting until an incident occurs to choose a fallback model is too late. Evaluate it beforehand on the same set and log and measure switches.

Going deeper

Anatomy of a retry storm

Retries multiply across layers. If an API gateway retries once, an application wrapper three times and the SDK twice by default, persistent failure delivers one user request to the API up to (1+1)×(1+3)×(1+2) = 24 times. During an incident, traffic can multiply many times over; failed requests also consume limits, delaying recovery. Backoff does not remove this multiplication.

Prevent storms in this order. First, reduce retries to one layer. Second, budget retries as a ratio: once retries exceed a share of normal requests, stop retrying and promptly mark the operations as failed. Third, use a circuit breaker and fallback when the dependency is unhealthy. Fourth, do not release the backlog all at once after recovery. Increase throughput gradually to avoid acceleration limits.

Validate recovery policies by injecting failures

Normal responses cannot reveal defects in retries or duplicate prevention. Build a failure-injection layer that inserts fake responses before model and tool calls, reproducing these situations.

There are three criteria: no business task executed twice, actual deliveries per request within budget, and final state labels matching reality. Put these checks in the deployment pipeline to catch regressions whenever retry logic changes.

At work

Clear Spring Water Purifiers: reports cut off at 60 seconds

The technician app at water-purifier rental company Clear Spring Water Purifiers generates customer water-quality history reports before visits. As reports grew longer, about 15% of requests failed. Technicians frequently pressed retry, generating the same report two or three times. Costs were 40% higher than expected from the number of reports.

Backend engineer Bong found nonstreaming model calls from the app server and a 60-second internal API-gateway timeout. Long reports took 70–90 seconds, so the gateway disconnected, but the app server's model request stayed alive, generating and billing through completion. The gateway, wrapper and SDK also retried separately.

Bong made these changes.

1. Switch report generation to streaming and disable buffering on that gateway route. Set timeouts to “30 seconds idle between events” and an overall four-minute deadline.
2. Keep retries only in the SDK; disable wrapper and gateway retries.
3. Close the upstream stream when a technician closes the screen.
4. Pregenerate reports for the next day's scheduled customers overnight. Generate only urgent same-day additions in real time.

Increasing the gateway timeout to 15 minutes was rejected because it tied up connections for longer and still left idle connections at risk of being closed by intermediate networks. Sending two identical requests simultaneously and taking the first answer was rejected for doubling costs. After the changes, failures fell below 1% and duplicate generation disappeared.

Pine Scent Publishing: a proofreading pipeline that does not restart from zero

Pine Scent Publishing splits book manuscripts by chapter, generates proofreading suggestions and saves them to its manuscript-management system. Repeated 529 errors halted overnight processing of 12 manuscripts and 380 chapters, so the operator restarted the job from the beginning that morning. Already-saved chapters were saved again, incurring two days' worth of processing costs. One chapter with many high-resolution scans repeatedly exceeded the size limit with 413 and kept blocking the retry queue.

Platform engineer Jin rebuilt the pipeline.

1. Store each chapter's state (pending, running, complete, failed, quarantined). Skip completed chapters on restart.
2. Use (manuscript ID, chapter number, proofreading round) as the save idempotency key. Return the existing result when that key recurs.
3. Classify errors with Chapter 4's decision table. Retry 529, 500 and rate-limit 429 within budget; send 413 and 400 directly to the dead letter queue without retrying. The operator splits quarantined chapters before resubmitting them.
4. Inject failures before launch. Terminate the process during work, then restart and verify no duplicate saves.

Moving everything to batch APIs was also considered. It would reduce cost, but editors begin reviewing chapters in order in the morning, while batches may take up to 24 hours and unfinished items can expire. They reserved batching for nonurgent repeat-proofreading rounds. Afterward, even on incident days, restarts handled only remaining chapters and no duplicate saves occurred.

Common mistakes

• Every 429 clears if you wait. Spend-limit or daily-quota 429 responses continue until limits reset or are adjusted. Distinguish them by body error code.
• Retrying faster and more frequently is better. Failed requests also consume limits. Use backoff, jitter, attempt/time caps and retry budgets.
• A 400 may work after a few retries. Request-format problems return the same result until fixed.
• A timeout means the server did not execute. The outcome is unknown. Look up modification state before executing again.
• Longer timeouts solve duplicate execution. They only reduce frequency. Idempotency keys address the cause.
• Make a new idempotency key on every retry. A changed key prevents recognizing the same operation. Use the same key for retries of one operation.
• Application retries on top of SDK retries are safer. Attempts multiply and amplify incidents. Retry in one layer only.

Summary at a glance

Check the official docs

• Anthropic — API errors: read error types, automatic SDK retries, request IDs and long-request advice.
• Anthropic — Rate limits: check token buckets, acceleration limits, spend-limit 429 and response headers.
• OpenAI — Error codes: check which specific 429 causes should be retried.
• OpenAI — Rate limits: read Retry-After, jitter and failed-request limit consumption.
• IETF — RFC 9110 §9.2.2: read idempotent methods and automatic-retry rules.
• AWS Builders' Library — Timeouts, retries, and backoff with jitter: read timeout settings, retry amplification and jitter.

Self-check

4-1. Explain why a modification request receiving no response must be treated as an unknown outcome rather than a failure, and list the recovery steps in order.

4-2. Name two pieces of information besides HTTP status needed to decide on automatic retries, and give one example of each changing the decision.

4-3. Explain how token-bucket limits differ from fixed-time resets, and why requests can be rejected even when the minute's total is within the limit.

4-4. Define connection timeout, idle timeout and overall deadline. Explain why idle timeouts are better than one request timeout for long generation.

4-5. State what an idempotency-key receiver (payment or booking server) must store, and how to handle the same key with different request content.

4-6. One LLM API supports real-time customer support, document summarization and nightly analysis. Design a table of retry budgets, concurrency limits, timeouts and incident behavior per feature.

Hands-on tasks