Triage and recover postmortems

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

Assume the bounded failure is active: analyze a supplied incident where a tokenizer upgrade doubled prompt size, exhausted worker memory, and caused 17 minutes of partial outage; reject hindsight claims unsupported by timestamped evidence. Begin with the user-visible objective—publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

jq '.events[] | {at,source,claim}' artifacts/timeline.json
./ops/postmortem contradictions INC-274

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.