Canary and operate postmortems

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

Before shipping Postmortems, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./ops/postmortem publish INC-274 --reviewers sre,ml-platform
./ops/actions follow INC-274 --until-closed

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: analyze a supplied incident where a tokenizer upgrade doubled prompt size, exhausted worker memory, and caused 17 minutes of partial outage; reject hindsight claims unsupported by timestamped evidence. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC.—as a regression scenario. The enduring production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.