Define the production target for postmortems

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

Start with the request or operator journey, not a tool choice. For Postmortems, draw POST /v1/summaries entering an AI document summarization service, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is analyze a supplied incident where a tokenizer upgrade doubled prompt size, exhausted worker memory, and caused 17 minutes of partial outage; reject hindsight claims unsupported by timestamped evidence. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Review every number against demand and consequence. deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

./ops/postmortem init INC-274 --template ops/postmortem.yaml
./ops/evidence list INC-274 --sort timestamp

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.