Design retries, degradation, and recovery

Map failures into invalid input, unauthorized action, transient dependency failure, permanent dependency rejection, internal defect, and ambiguous outcome. Retry only transient failures, with exponential backoff, jitter, a strict attempt cap, and an overall deadline shorter than the caller’s timeout. Invalid signatures, authorization failures, and schema errors are not retryable. An ambiguous side effect requires an idempotency lookup before another attempt.

The known hazards are low canary volume, cache warming bias, unequal request mixes, shared dependency failures, and automation promoting on missing data. Pick three and inject them locally. For each, record expected HTTP behavior, durable state, user-facing response, alert, and operator action. A handled failure is not merely caught; it leaves the system in a state an operator can explain and recover.

Define a degraded mode that removes risky capability while preserving honest service. Never convert unavailable evidence into a confident answer. Return a typed temporarily_unavailable, serve a known-safe control, accept to a durable inbox, or create a human handoff as appropriate. Include a retry-after hint only when the service can support it.

Use a circuit breaker around a failing dependency to prevent retry storms. The breaker should open on a rolling failure threshold, permit a small number of probes after a cool-down, and emit state transitions. Bulkhead concurrency so one slow provider cannot consume every worker. Queue consumers need maximum attempts and a dead-letter destination with a replay procedure.

Execute the operational rollback: route zero traffic to v2, preserve its logs and image digest, and confirm v1 capacity before terminating v2. Measure time from the first injected breach to restored safe behavior. Confirm new requests use the safe path, in-flight requests are drained or completed according to contract, and no durable work is lost. Preserve the failed revision’s logs, configuration, and fixtures for diagnosis.

After rollback, reconcile ambiguous records. Compare ingress IDs with completion or outcome IDs, identify duplicates and gaps, and repair using an idempotent replay command. The replay command must support dry-run, bounded batches, and an operator-supplied reason. Never repair production by manually editing rows without an audit record.

This chapter passes when every classified failure has a deterministic disposition, retries are bounded, degradation is truthful, and rollback is rehearsed rather than merely documented. Save timestamps and commands from the drill. If recovery requires the original author’s memory, the system is not operable.

Make the disposition executable. This handler converts canary_guardrail_breached into a bounded response and never retries a policy or contract failure. The reason is safe for logs; detailed dependency text stays in a protected trace.

from dataclasses import dataclass

@dataclass(frozen=True)
class Recovery:
    action: str
    retryable: bool
    retry_after_seconds: int | None

def recover(kind: str, attempts: int) -> Recovery:
    if kind == "canary_guardrail_breached":
        return Recovery("traffic.v1=100 traffic.v2=0", False, None)
    if kind == "transient_dependency" and attempts < 3:
        return Recovery("retry_with_jitter", True, 2 ** attempts)
    return Recovery("hold_and_escalate", False, None)

Exercise the branch against the local service and capture both the control action and request identity. The expected result proves safe behavior without erasing evidence.

$ curl -sS -H 'X-Debug-Failure: canary_guardrail_breached' http://localhost:8000/answer
{"status":"degraded","action":"traffic.v1=100 traffic.v2=0","retryable":false}
$ curl -sS http://localhost:8000/answer
{"status":"ok","request_id":"fixture-001","revision":"stable"}

A generic 500 means classification is incomplete; do not add broad retries. If recovery says success but the second response still names the failed revision, routing or cache invalidation is incomplete. Preserve the identity, old configuration digest, and rollback timestamp before replaying ambiguous work.