API errors & retries
Handle failures and retries
When 429 storm with missing Retry-After that would amplify to 40 calls, the system must degrade on purpose without widening blast radius.
1Learn the idea
Read
Classify and bound retries
Map failure classes for POST /v1/chat: retryable vs fatal vs needs-human. Retries need budgets, jitter, and idempotency rules aligned to attempts ≤ 3, wall budget ≤ 8s, non-idempotent POSTs never replay without Idempotency-Key. The chapter’s signature failure — 429 storm with missing Retry-After that would amplify to 40 calls — must take a deliberate branch, not a generic catch-all.
Read
Containment path
Implement the degrade/rollback/refuse behavior mobile client waiting on a grounded answer needs when INC-429-2026-07-11 repeats. Prefer scoped controls (one flag, one weight, one tenant, one secret version) over fleet-wide restarts. Preserve evidence; do not delete logs to “clean the demo.”
Read
Implementation artifact
async function retryingFetch<T>(url: string, init: RequestInit, gate: Gate): Promise<FetchResult<T>> {
let attempts = 0;
const deadline = Date.now() + gate.wallMs;
while (attempts < gate.maxAttempts && Date.now() < deadline) {
attempts += 1;
const res = await fetch(url, { ...init, headers: { ...init.headers, "Idempotency-Key": init.headers?.["Idempotency-Key"] ?? crypto.randomUUID() } });
if (res.status === 429) {
const wait = parseRetryAfter(res.headers.get("Retry-After")) ?? jitter(attempts);
if (Date.now() + wait >= deadline) return { ok: false, code: "budget", attempts };
await sleep(wait); continue;
}
if (res.status === 401) return { ok: false, code: "auth", status: 401, attempts };
if (!res.ok && gate.retryable.has(res.status)) { await sleep(jitter(attempts)); continue; }
if (!res.ok) return { ok: false, code: "http", status: res.status, attempts };
try { return { ok: true, data: await res.json() as T, attempts, latencyMs: Date.now() }; }
catch { return { ok: false, code: "invalid_json", attempts }; }
}
return { ok: false, code: "budget", attempts };
}
Read
Verify harm reduction
After containment, check retry_attempts_total{outcome} and p95 wall_ms ≤ 8000 moves in the safe direction and watch for retry amplification. Write the stop condition that ends the incident response for this lab.
Read
Stage depth
Chaos note: inject only one fault class at a time and restore fixtures after. Watch for dual failures — dependency down and retry amplifier — which is how 429 storm with missing Retry-After that would amplify to 40 calls becomes an outage. Customer communication templates (even if only for the drill) beat silence. If you queue deferred work, define poison-message handling. Budget documents should state the maximum extra spend allowed during retries. Close the loop by linking the containment action to a dashboard panel for retry_attempts_total{outcome} and p95 wall_ms ≤ 8000.
Read
Field notes for `api-error-handling` / `failure-handling`
Draw a state diagram for degrade modes and put it in the repo as ASCII if needed. Cap concurrent retries across the process, not only per request. Ensure cancellation propagates to downstream HTTP clients. When failing closed, choose a user-visible message that does not leak internals. Practice the single command that flips the kill switch or weight to zero. After recovery, drain or inspect deferred work before declaring green. In this chapter the product is resilient TypeScript fetch wrapper around POST /v1/chat, the human stakeholder is mobile client waiting on a grounded answer, and the incident id you design against is INC-429-2026-07-11. Re-state the oracle in your notes — 429 + Retry-After:2 → sleep once → 200 with attempts=2 — and keep the invariant visible: attempts ≤ 3, wall budget ≤ 8s, non-idempotent POSTs never replay without Idempotency-Key. Track retry_attempts_total{outcome} and p95 wall_ms ≤ 8000 as the scoreboard. Surface under change control: POST /v1/chat. If you only have forty minutes, finish the fixture for 429 storm with missing Retry-After that would amplify to 40 calls before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.
Go deeper
Before you start
Why this matters
Assume 429 storm with missing Retry-After that would amplify to 40 calls is happening right now. Write the first safe action, the signal that confirms containment, and the action you will not take (infinite retry, broad restart, deleting evidence). Tie the plan to invariant: attempts ≤ 3, wall budget ≤ 8s, non-idempotent POSTs never replay without Idempotency-Key.
In the wild
See how this idea shows up as a product and a company — then come back to the lesson. Skills transfer across vendors.
Related lessons
Check your understanding
Page assessment
Answer from memory. Completion is saved from this evidence, not from opening the next page.
All responses are required.