Inject a controlled postmortems failure

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

This page deliberately breaks Postmortems in a disposable or explicitly approved environment. The scenario is analyze a supplied incident where a tokenizer upgrade doubled prompt size, exhausted worker memory, and caused 17 minutes of partial outage; reject hindsight claims unsupported by timestamped evidence. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed and record the effective configuration:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

./ops/replay tokenizer-v4 --fixture fixtures/large-doc.json --memory 1Gi
./ops/metrics compare --before v3 --after v4 prompt_tokens container_oom_events_total

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC. The production tradeoff is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.