Chapter 5. Infrastructure Reliability, Secrets Management, and Incident Response

Learning objectives

• Design a dual-key (overlapping validity) acceptance architecture to rotate JWT signing secrets, webhook secrets, API keys, and TLS certificates with zero downtime, and execute emergency revocation when a key is leaked.
• Compare periodic snapshot backups against WAL-based Point-in-Time Recovery (PITR) in terms of Recovery Point Objective (RPO) and Recovery Time Objective (RTO), and verify backup readiness through regular restore drills.
• Design a three-to-four phase Expand-Contract (Parallel Change) zero-downtime database schema migration so old and new application versions can coexist during rolling deployments, while avoiding destructive table locks.
• Trace end-to-end latency bottlenecks from the edge through the database and external LLMs using OpenTelemetry metrics, structured logs, and distributed traces (trace_id), while scrubbing sensitive secrets and personal data from telemetry pipelines.
• Combine soft anomaly alerts, hard spending/rate caps, exponential backoff with jitter, and circuit breakers to prevent retry storms and runaway cloud/AI bills.
• Execute incident isolation and rollback runbooks under the "mitigate first, investigate later" principle using instant rollbacks and feature flags, and conduct blameless postmortems that produce concrete engineering action items.

Introduction

There is a time-tested proverb among operations engineers: "Every system eventually fails, every secret key eventually needs to be rotated, and an untested backup is not a backup at all."

The first operational crucible is secret rotation. Security policy requires rotating your session signing secret and payment webhook verification key every quarter—yet when an engineer simply overwrites the environment variable with a new key and redeploys, tens of thousands of active users are forcibly logged out at once, or half of incoming payment webhooks fail signature checks (401) during the three-minute rolling deployment window. Even more urgent is the moment a developer accidentally pushes an .env file containing a live API key to a public GitHub repository, forcing you to replace and revoke the key in minutes.

The second operational crucible is database recovery and schema evolution. At 2:17 p.m. on a Tuesday, a bug in a newly deployed batch script runs an UPDATE statement without a WHERE clause and overwrites every customer's subscription tier. If your only backup is a daily snapshot taken at midnight, restoring from that snapshot wipes out 14 hours of legitimate customer orders and exam submissions recorded between midnight and 2:16 p.m. Similarly, if an engineer runs ALTER TABLE RENAME COLUMN directly on a live table during a rolling deployment, every server instance still running the previous application version immediately crashes with column does not exist errors.

This chapter teaches the engineering practices that keep a live production service running without a single second of unnecessary downtime: rotating secrets and evolving database schemas safely, recovering from catastrophic data corruption to the exact second before the mistake with PITR, illuminating distributed bottlenecks with OpenTelemetry while scrubbing sensitive data from logs, and executing a disciplined five-minute mitigation runbook when the pager goes off at 3 a.m.

Key concepts

Zero-downtime secret and certificate rotation: The dual-key acceptance pattern

Why does rotating a session cookie signing secret (SESSION_SECRET), an internal service-to-service token, an incoming webhook signing secret, or a database password so often cause dropped requests and forced logouts?

Because in a naive Single-Key implementation, the application holds only one key (SECRET_KEY) at a time:

1. At 10:00 a.m., users sign in and receive session cookies signed with the old key (Key_V1).
2. At 10:10 a.m., an administrator overwrites the server environment variable with a brand-new key (Key_V2) and triggers a rolling deployment.
3. While the rolling deployment is underway, half of your server instances are still checking Key_V1 while the other half only recognize Key_V2.
4. Once the deployment finishes, every returning user whose browser presents a cookie signed with Key_V1 is rejected by the Key_V2-only servers as an "invalid signature"—logging out your entire user base simultaneously! Likewise, for incoming webhooks, because you cannot synchronize the external webhook sender's key switch and your server's redeployment down to the same millisecond, a single-key swap guarantees failed webhook deliveries during the transition window.

The standard architectural solution is the Dual-Key Acceptance pattern (also called overlapping validity or a multi-key ring with CURRENT + PREVIOUS slots). The core principle is simple: "Always sign (issue) new tokens using only the newest key, but verify (accept) incoming tokens against an ordered list containing both the newest key and the immediately preceding key during a grace period."

// Example of a Dual-Key (key ring) verification setup for zero-downtime rotation
// Injected as SESSION_SECRETS="v2_new_secret_key,v1_old_secret_key"
const keyRing = (process.env.SESSION_SECRETS ?? "")
  .split(",")
  .map((k) => k.trim())
  .filter(Boolean);

export function signToken(payload: string): string {
  // Newly issued tokens are ALWAYS signed exclusively with the first (newest) key: keyRing[0].
  return hmacSign(payload, keyRing[0]);
}

export function verifyToken(token: string): boolean {
  // Verification tries the newest key first, and falls back to the previous key (keyRing[1])
  // so existing sessions and in-flight webhooks remain valid during the rotation grace window.
  return keyRing.some((secret) => hmacVerify(token, secret));
}

With dual-key verification built into your code, you can rotate any signing secret or webhook key with zero dropped requests by following a four-step rotation runbook.

Rotating database credentials with zero downtime follows the exact same principle using the Alternating Users (Blue/Green DB Users) strategy: create two database roles (app_user_blue and app_user_green) with identical permissions. While the live application is connected as app_user_blue, rotate the password on the idle app_user_green role, deploy the application with its connection string updated to app_user_green, and once all old blue connections have drained, lock or rotate app_user_blue.

Scheduled rotation vs. emergency revocation during an active key leak

What if a rotation is not a routine quarterly exercise, but an emergency response to an active compromise—such as a developer accidentally pushing a live .env file to a public GitHub repository? Should you still keep the leaked V1 key active for a 24-hour grace period?

No! In an active leak, your response depends on the type of key compromised:

1. External API keys and cloud credentials where your server is the caller (such as an OpenAI API key, AWS IAM key, or Supabase service_role key):
   • Immediately generate a replacement key (V2) in the vendor console, inject V2 into your production environment variables, and trigger a fast deployment (1–2 minutes) so legitimate traffic switches to V2. The instant V2 is live, permanently revoke (delete) the leaked V1 key with zero grace period. If attackers are already actively draining data or racking up thousands of dollars per minute on V1, revoke V1 immediately even before V2 finishes deploying—a brief 60-second service error is vastly preferable to a total database breach.
2. Session signing secrets (SESSION_SECRET) leaked to an attacker:
   • Because anyone holding the signing secret can forge arbitrary administrator session cookies locally, leaving V1 in the verification key ring would continue accepting attacker-forged cookies. In a signing-key compromise, you must skip the grace period, remove V1 immediately, and invalidate all existing sessions—accepting the inconvenience of forcing all users to sign in again in order to lock out the attacker.

Backup strategies and Point-in-Time Recovery (PITR): Designing RPO, RTO, and restore drills

Every disaster recovery plan starts by defining two quantitative time targets:

• RPO (Recovery Point Objective): "How much historical data (measured in time before the incident) can the business afford to lose?" An RPO of 24 hours means losing up to a full day of recent writes is acceptable; an RPO of 1 minute means you must be able to recover database state up to within 60 seconds before the failure occurred.
• RTO (Recovery Time Objective): "Once an outage or data corruption occurs, how quickly (measured in elapsed time after the incident) must we restore the service to normal operation?"

Database backup mechanisms fall into two tiers—Periodic Snapshots (daily dumps) and Point-in-Time Recovery (PITR)—and the RPO they achieve differs dramatically.

For any production database storing payments, exam records, certificates, or customer workspaces, enabling WAL-based PITR is mandatory.

However, flipping the PITR toggle in a dashboard is only half the job. Countless teams discover during a real crisis that (1) their backup files were silently failing or corrupted, (2) no one on call has the KMS decryption permissions needed to restore the snapshot, (3) restoring a 500 GB database takes 6 hours and blows past their 30-minute RTO target, or (4) restoring the database leaves object storage (storage.objects) out of sync. Therefore, mature infrastructure teams run a scheduled Restore Drill at least once a quarter: restoring a production PITR backup into an isolated staging instance, running automated data-integrity verification queries, and recording the actual wall-clock time required to complete the restore.

Zero-downtime schema migrations: The Expand-Contract (Parallel Change) pattern

Modern cloud applications deploy via rolling updates or canary releases without taking down the site. That means during every deployment—and whenever you need to roll back—instances running the old application code (App v1) and instances running the new application code (App v2) are alive at the exact same time, reading from and writing to the exact same database.

If a developer forgets this concurrency reality and tries to rename the full_name column on the users table to display_name in a single step by running:

-- STRICTLY FORBIDDEN IN ZERO-DOWNTIME PRODUCTION!
-- Immediately breaks running v1 app instances and makes rollback impossible.
ALTER TABLE users RENAME COLUMN full_name TO display_name;

The instant that ALTER TABLE RENAME COLUMN statement commits, every running App v1 instance that executes SELECT full_name FROM users crashes with a 500 Internal Server Error! Even after App v2 finishes rolling out, if you discover a critical bug in App v2 ten minutes later and click Rollback to revert to App v1, App v1 is now completely incompatible with the mutated database schema—trapping you in a one-way door where rollback is impossible.

The universal engineering pattern that solves this problem is the Expand-Contract (or Parallel Change) pattern, which breaks a destructive schema change into three to four backward-compatible phases.

When executing DDL statements against large production tables in PostgreSQL, you must also guard against table locks that can freeze live traffic:

• A standard CREATE INDEX acquires a lock that blocks all concurrent writes (INSERT, UPDATE, DELETE) on the table until the index build finishes. On live production tables, always use CREATE INDEX CONCURRENTLY, which builds the index without blocking writes (note that CONCURRENTLY cannot run inside a BEGIN ... COMMIT transaction block, so configure your migration runner to execute it outside a transaction).
• When adding a CHECK or FOREIGN KEY constraint, avoid locking the table for a full scan by adding the constraint with ADD CONSTRAINT ... NOT VALID first (which updates metadata instantly and enforces the rule on new writes) and then running VALIDATE CONSTRAINT in a separate step (which scans existing rows using a weaker lock that does not block concurrent reads or writes). Finally, always start migration sessions with SET lock_timeout = '3s' so that if a lock cannot be acquired immediately, the migration fails safely instead of queuing up and blocking all application queries.

The three pillars of observability and OpenTelemetry distributed tracing

When an incident strikes, there is a world of difference between guessing "the site feels slow today" and observing "request trace_id=4bf92f... entered the Seoul function, acquired a Supavisor connection in 2 ms, finished its PostgreSQL query in 4 ms, and stalled for 28.4 seconds inside the outbound LLM API span." The engineering discipline of understanding a system's internal state from its external telemetry outputs is Observability, built on three complementary pillars.

When monitoring latency metrics, relying on the arithmetic mean (Average Latency) is a classic trap. If 99 static or cached requests finish in 10 ms and 1 AI generation request hangs for 20,000 ms (20 seconds), the average latency across all 100 requests is a healthy-looking 210 ms—completely hiding the fact that users triggering the AI workflow are experiencing a 20-second stall. Production Service Level Objectives (SLOs) and alert thresholds must always track tail latency percentiles (p95 and p99).

Log scrubbing and redacting sensitive information

An ironclad security rule when building an observability pipeline is Log Scrubbing (Redaction): never allow secret credentials or raw personal data to be written into logs or trace span attributes. If a developer dumps full HTTP request headers, raw body payloads, or process.env into console.log or OpenTelemetry spans for easier debugging, sensitive values are shipped to third-party log aggregation platforms (Datadog, Sentry, CloudWatch) and stored in plaintext indefinitely:

• Authorization: Bearer ... headers and session cookies (Cookie, Set-Cookie)
• LLM and cloud API keys (sk-...), and database connection strings containing passwords (postgresql://user:password@...)
• User plaintext passwords, national ID numbers, payment card numbers, and confidential user prompt contents

To prevent your logging platform from becoming a data breach vector, configure automated redaction filters (scrubbers) in both your application logger and your OpenTelemetry Collector to replace values matching keys like password, secret, token, authorization, cookie, and api_key with [REDACTED] before telemetry ever leaves the process, and correlate user issues using a non-reversible hashed user identifier alongside trace_id.

Cost and usage anomaly alerts and circuit breakers

Cloud serverless functions and external LLM APIs operate on utility-style pay-as-you-go billing. When a traditional fixed-capacity server was hit by a traffic flood or an infinite loop bug, the server simply maxed out its CPU and stopped responding; in an autoscaling cloud environment wired to paid LLM APIs, an infinite retry loop in an agent or a targeted Denial of Wallet attack can burn tens of thousands of dollars overnight while everyone is asleep.

Protecting both system availability and financial survival therefore requires pairing Soft Limits (alerts) with Hard Limits (automated enforcement) and resilience patterns:

1. Anomaly Detection and Budget Alerts (Soft Limits): Configure billing and token monitors to page the on-call channel not only at 50%, 80%, and 100% of your daily budget, but whenever the hourly spend velocity spikes above 3× the normal baseline.
2. Hard Spending Caps and Per-Tenant Rate Quotas (Hard Limits): Because an email alert at 3:00 a.m. will not stop an automated runaway loop by itself, enforce programmatic daily token quotas per user/tenant and hard monthly spend ceilings at the AI Gateway and provider levels—automatically rejecting excess requests (429) or gracefully degrading to a cheaper, smaller model when thresholds are crossed.
3. Exponential Backoff with Jitter and Circuit Breakers: When a downstream database or external LLM API is struggling under heavy load, having hundreds of serverless instances hammer it with immediate retries every 100 milliseconds triggers a Retry Storm (Thundering Herd) that knocks the recovering dependency completely offline. Always space out retries exponentially (1 s → 2 s → 4 s) with randomized jitter so retries do not synchronize, and wrap external calls in a Circuit Breaker that trips Open once consecutive failures cross a threshold—immediately returning a fast fallback response without calling the overloaded dependency until its cooldown timer expires.

Incident isolation/rollback runbooks and blameless postmortems

When a P1 production outage erupts, the single most destructive mistake an on-call engineer can make is saying: "Let me read through the code line by line to figure out why this bug is happening, write a fix, run tests, and deploy a patch"—leaving live customers broken for an hour while debugging!

The cardinal rule of incident response is "Mitigate First, Investigate Later — stop the bleeding before diagnosing the disease."

A standard production incident runbook executes five steps in strict order:

1. Detect & Triage: When an SLO burn alert fires (for example, 5xx error rate exceeding 5% over 5 minutes), assign an Incident Commander (IC) immediately and classify severity (P1 total outage vs. P2 partial feature degradation).
2. Isolate & Mitigate (target: under 5 minutes):
   • If the incident started shortly after a code or configuration deploy: Do not debug the code live; execute an Instant Rollback to the last known-good deployment (such as Vercel Instant Rollback) immediately.
   • If a specific new feature or non-critical integration is failing: Flip its Feature Flag (Kill Switch) off to isolate the faulty component while keeping the core service healthy.
   • If an external LLM provider is down: Force AI Gateway fallback routing to the secondary provider; if under abusive traffic attack, tighten WAF rate limits or challenge rules.
3. Communicate: Post clear, regular status updates to internal stakeholders and the public customer Status Page describing user impact and ongoing mitigation.
4. Resolve & Verify: Once customer impact is mitigated and the system is stable, use logs, distributed traces (trace_id), and metrics to identify the root cause, write a regression test, and deploy the verified fix.
5. Blameless Postmortem: After the incident is closed, if leadership asks "Whose fault was this?" and punishes the individual who merged the PR, engineers will hide mistakes in the future. A Blameless Postmortem instead asks: "What gap in our automated checks, deployment safeguards, or architecture allowed a human mistake to reach production and cause an outage?" It documents an exact minute-by-minute timeline, conducts a 5 Whys root-cause analysis, and assigns concrete, deadline-tracked engineering action items (such as adding a CI linter that blocks destructive DROP COLUMN migrations or adding a circuit breaker).

Going deeper

A zero-lock DDL checklist for operating PostgreSQL at scale

Even when following the Expand-Contract pattern, running a seemingly innocent ALTER TABLE statement on a high-traffic PostgreSQL table can freeze your entire application for several minutes if you do not account for PostgreSQL's lock queue mechanics.

Why can a 1-millisecond metadata operation like ALTER TABLE users ADD COLUMN bio text; bring down a busy site? Because ALTER TABLE requires a brief ACCESS EXCLUSIVE lock—the strictest lock mode in PostgreSQL, which conflicts with every other operation on that table, including ordinary SELECT reads.

Now trace what happens if a 50-second analytical reporting query is already running on users when your migration executes ALTER TABLE:

1. The slow reporting query holds a lightweight ACCESS SHARE read lock on users.
2. Your ALTER TABLE statement cannot acquire ACCESS EXCLUSIVE until the reporting query finishes, so ALTER TABLE enters the lock wait queue.
3. Here is the catastrophe: Thousand of ordinary user SELECT and INSERT queries arriving milliseconds later could normally run alongside the reporting query without any conflict—but because an ACCESS EXCLUSIVE request (ALTER TABLE) is waiting ahead of them in the queue, every single incoming SELECT and INSERT is blocked behind the waiting ALTER TABLE! As a result, a 1 ms schema migration waiting behind a 50-second slow query causes a 50-second total outage across your entire application (Lock Queue Pile-up).

To prevent lock queue pile-ups in production, every PostgreSQL migration script must enforce the following safeguards:

-- 1) Cap lock wait time strictly so that if the lock cannot be acquired within 3 seconds,
-- the migration aborts cleanly and retries later instead of blocking live user queries!
SET lock_timeout = '3s';
SET statement_timeout = '30s';

-- 2) Add new foreign keys or CHECK constraints as NOT VALID first (instant metadata update),
ALTER TABLE orders
  ADD CONSTRAINT orders_amount_positive CHECK (amount >= 0) NOT VALID;

-- 3) Then validate existing rows in a separate step without blocking concurrent reads/writes.
ALTER TABLE orders
  VALIDATE CONSTRAINT orders_amount_positive;

Tiered health checks and graceful degradation in multi-region, multi-vendor AI services

In AI applications that depend on multiple external APIs, preventing a failure in a secondary convenience feature from taking down the core product (the Bulkhead pattern) is a hallmark of senior reliability engineering.

Consider an exam platform where candidates take timed certification exams (the critical path) and can also view an AI-generated study guide for missed practice questions (a non-critical enrichment feature) hosted on the same service. What happens if your load balancer's health check endpoint (/api/health) pings the external LLM provider and returns 503 Service Unavailable whenever the LLM API is down? During an OpenAI or Anthropic outage, your load balancer marks all of your application servers as dead—taking down user sign-in, exam taking, and score reports across the entire site just because the optional AI study guide generator is unreachable!

To prevent self-inflicted outages, strictly separate Hard Dependencies (Critical Path) from Soft Dependencies (Non-critical Path):

• Liveness and Readiness Health Checks queried by Route 53 or your load balancer must test only the local application process and hard dependencies required to serve core traffic (such as primary database connectivity).
• When a soft or external dependency (an LLM API, a recommendation engine, or an email delivery service) fails or trips its circuit breaker, keep the server healthy and switch into Graceful Degradation: serve the core exam and scoring flows at 100% availability while replacing the AI commentary widget with a friendly notice ("AI study guide generation is temporarily delayed and will be queued to your account shortly").

Workplace scenarios

Onbit Pay: Rotating session signing keys with zero forced logouts while revoking a leaked external API key in minutes

While preparing for an annual financial security audit, AI corporate expense-management platform Onbit Pay faced two urgent tasks on the same afternoon. First, they needed to rotate their 12-month-old user session JWT signing secret (AUTH_SECRET) and corporate card webhook verification secret without logging out 3,000 active finance managers or dropping live card-swipe webhooks. Second, at 3:00 p.m. that same day, an automated secret scanner alerted them that an external contractor had accidentally pushed a test script to a public GitHub Gist containing the live production API key for their external receipt-OCR AI vendor.

Recognizing that a scheduled internal key rotation and an active public key leak require completely different procedures, the platform security lead directed two parallel playbooks:

1. Leaked external OCR AI API key → Emergency Revocation playbook (completed in 4 minutes):
   • Because the key was already exposed on the public internet, waiting for a grace period was out of the question. The team immediately generated a new API key (Key_New) in the OCR vendor's console, updated the Vercel Production environment variable, and redeployed (taking 2 minutes) so live receipt scans switched to Key_New. Seconds later, they permanently revoked (Revoke) the leaked old key (Key_Old) in the vendor console. Auditing 18 minutes of AI Gateway logs between the Gist push and revocation confirmed zero unauthorized external calls had occurred.
2. Session signing secret & card webhook secret → Dual-Key zero-downtime rotation playbook:
   • To ensure zero dropped card webhooks and zero forced logouts across 3,000 active users, they executed the four-step dual-key rotation workflow.
   • Step 1: They deployed code accepting an array of [New_Key, Old_Key] for signature verification. Step 2: They switched new session token signing and the card issuer's webhook configuration to use New_Key. Step 3: Over the 24-hour session expiration window, they monitored structured logs until verification hits on Old_Key dropped to zero. Step 4: They removed Old_Key from the environment variable. The entire rotation completed with zero forced logouts and zero missed webhooks.

Haedam Clinic: Rewinding overwritten appointment records by three minutes with PITR and migrating safely with Expand-Contract

At Haedam Clinic, which operates a telemedicine appointment booking service, a deployment went out on Tuesday at 3:12 p.m. to rename the doctor_name column on the appointments table to practitioner_id and clean up expired booking rows. Unfortunately, the developer made two catastrophic mistakes in the migration script: first, they ran ALTER TABLE appointments DROP COLUMN doctor_name while old application instances were still serving traffic, causing immediate 500 errors across half the fleet; second, a data-cleanup SQL statement omitted its WHERE status = 'expired' clause, and at 15:12:10 overwrote the status of all 14,000 upcoming patient appointments to 'cancelled'!

When the P1 alert fired, the Incident Commander executed the incident runbook step by step.

Common mistakes

1. "To rotate a secret key with zero downtime, just overwrite the existing environment variable in the dashboard with the new key and redeploy." Overwriting a single key in place causes verification failures across server instances during the rolling deployment and immediately invalidates all existing user sessions and in-flight webhooks signed with the old key. Zero-downtime rotation requires a Dual-Key acceptance window where [New_Key, Old_Key] are both accepted for verification while only New_Key is used for signing new tokens.
2. "Because we take an automated full database snapshot every night at 3:00 a.m., we can recover from an accidental DELETE at 2:00 p.m. with zero data loss." A daily snapshot alone leaves up to a 24-hour RPO gap: restoring the 3:00 a.m. snapshot wipes out all 11 hours of legitimate customer transactions written between 3:00 a.m. and 2:00 p.m. Recovering to the exact second before an afternoon human error requires continuous Write-Ahead Log (WAL) archiving with Point-in-Time Recovery (PITR) and regular restore drills.
3. "Because our platform supports zero-downtime rolling deployments, it is safe to rename a database column directly with ALTER TABLE RENAME COLUMN." During a rolling deployment, old and new application instances run side by side against the same primary database. Renaming or dropping a column in a single step immediately breaks running old instances and makes rolling back impossible; you must use the four-phase Expand-Contract (Parallel Change) pattern (Expand → Dual-Write & Backfill → Cutover → Contract).
4. "Logging the full raw HTTP headers and body of every request to our central logging service is the best way to speed up debugging." Raw request headers and bodies contain Authorization bearer tokens, session cookies, plaintext passwords, API keys, and personal data—turning your central log store into a severe credential and privacy breach. Always scrub and redact sensitive fields before logs or OpenTelemetry spans leave the process, and correlate requests using trace_id.
5. "Our service's Average Latency is 200 ms, which proves every user is getting a fast experience." When fast static or cached responses make up 95% of traffic, the arithmetic mean masks severe 20-second stalls experienced by users on AI generation or database-heavy routes. Production reliability SLOs and alerts must always track tail latency percentiles (p95 and p99).
6. "When a P1 outage occurs right after a deployment, you should keep the new version live while you read the logs, find the root cause, and write a bug-fix commit." Spending an hour debugging live while customers are failing violates the cardinal rule of incident response ("Mitigate first, investigate later"). Always mitigate customer impact first (via instant deployment rollback, feature flag kill switches, or AI Gateway fallbacks) to restore service in minutes, and conduct root-cause analysis and a blameless postmortem afterward.

Summary at a glance

Check the official docs

• Supabase — Database Backups & Point-in-Time Recovery (PITR): How daily physical backups compare to WAL-based PITR in RPO and how point-in-time restoration works.
• PostgreSQL Documentation — Routine Database Maintenance & Concurrently: Building indexes without blocking writes using CREATE INDEX CONCURRENTLY and understanding lock levels.
• OpenTelemetry Documentation — Context Propagation & Collector Scrubbing: The metrics, logs, and traces signal standards and redacting sensitive attributes in the Collector.
• OWASP — Secrets Management Cheat Sheet: Secret lifecycle management and designing automated zero-downtime rotation.
• NIST — SP 800-61 Rev. 3 Incident Response Recommendations: Standard phases for incident detection, containment/mitigation, recovery, and post-incident lessons learned.

Self-check

5-1. Describe in order the four steps of the Dual-Key acceptance pattern for rotating a webhook signing secret or session signing key with zero downtime and zero dropped requests, and contrast why an external API key accidentally pushed to a public GitHub repository must instead be revoked immediately without a grace period.

5-2. Define Recovery Point Objective (RPO) and Recovery Time Objective (RTO), explain why a database backed up only via a daily 04:00 a.m. snapshot cannot recover without data loss when an accidental DELETE query runs at 2:00 p.m., and name the backup mechanism required to solve this problem.

5-3. Explain why executing ALTER TABLE RENAME COLUMN directly on a live table in a rolling-deployment environment causes immediate errors and blocks application rollback, and list in order the four phases of the Expand-Contract (Parallel Change) pattern that solve this problem.

5-4. Explain why observability dashboards and SLOs must track p95 and p99 percentile latency rather than Average Latency, and list at least three categories of sensitive data that must be scrubbed (redacted) when collecting structured logs and distributed traces (trace_id).

5-5. Describe what happens (the Retry Storm / Thundering Herd phenomenon) when a downstream database or external LLM API experiences transient overload and every application server immediately retries failed calls in a tight loop, and name both the retry spacing technique and the traffic-cutoff pattern used to prevent cascading failure.

5-6. Design task: At 02:00 a.m., immediately after a new production deployment, a P1 alert fires because the payment API's 500 error rate has spiked to 15%. When the on-call engineer says, "I will read the logs, debug the code, and write a bug-fix patch," explain why the Incident Commander must stop them, state the standard mitigation actions to take within the first 10 minutes, and describe the core principle of the postmortem written after the incident is resolved.

Hands-on tasks