Postmortem lab
Configure the postmortems contract
Production rule: Configure the runtime contract for an AI document summarization service; no stage is complete until another operator can reproduce its evidence and reverse its risky action.
Before you start
Why this matters
In two minutes, write the user-visible outcome this page protects, one numerical threshold, and the first signal you expect to move. Then name an observation that would prove your initial theory wrong. Keep the answer beside your terminal; this lab rewards prediction before inspection rather than explanations invented after the graph changes.
1Learn the idea
Read
Lab target
You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.
Read
Build a reproducible environment
The contract for Postmortems must survive a clean checkout. Pin tool and image versions, provide sample inputs, and fail startup on malformed configuration. The service boundary is POST /v1/summaries on an AI document summarization service. Before starting dependencies, list required ports, identities, secrets, storage, and cleanup steps. Use synthetic credentials and redacted fixtures; production tokens do not make a staging lab more realistic.
Create the configuration as a reviewed artifact:
postmortem:
incident_id: INC-274
impact_window_utc: [14:03, 14:20]
sections: [impact, detection, timeline, causal_analysis, response, actions]
actions_require: [owner, due_date, verifier, evidence]
prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5
For every field, document type, valid range, default, reload behavior, and failure behavior. Numbers are incomplete without units. A timeout of 40 could mean milliseconds or seconds; a ratio of 0.005 could be a fraction or percentage. Reject unknown keys so a typo cannot silently select a permissive default. If live reload is supported, expose a configuration revision in telemetry and keep the previous valid revision for rollback.
Read
Verify interfaces and state
Run the setup workflow:
./ops/timeline merge artifacts/logs.json artifacts/deployments.json --timezone UTC \
> artifacts/timeline.json
./ops/timeline validate artifacts/timeline.json
The expected result is not merely exit code zero. Confirm the process identity, dependency reachability, effective configuration, and one health or introspection response. Then restart the component and prove durable state remains correct. Run the setup twice to expose non-idempotent creation, conflicting ports, or duplicate policy entries. Capture the exact versions in the evidence bundle.
The production objective remains publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. Therefore setup must expose container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed before meaningful traffic arrives. Query each metric by its stable labels and verify an idle system is distinguishable from a broken scraper. Generate one event and check that the relevant counter changes by exactly one. Reject high-cardinality labels and ensure sensitive configuration values are redacted from startup logs.
Read
Contract review and rollback
Model a bad configuration: remove one required field, set one threshold just outside its range, and reference an unavailable dependency. Each case should fail predictably, explain the field, and avoid partial state. Next, apply the previous good revision and time recovery. A rollback that requires undocumented shell history is not a rollback plan.
The central design tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. Record the chosen default, its operational owner, and a date for reevaluation. Complete setup only after a teammate can run it from the repository instructions, produce the same effective configuration, and cleanly tear it down without affecting shared environments.