Chapter DPostmortem labPage 3 of 8

Postmortem lab

Run the postmortems baseline

Production rule: Code and measure the normal path for an AI document summarization service; no stage is complete until another operator can reproduce its evidence and reverse its risky action.

~30 minHappy path

Before you start

Why this matters

In two minutes, write the user-visible outcome this page protects, one numerical threshold, and the first signal you expect to move. Then name an observation that would prove your initial theory wrong. Keep the answer beside your terminal; this lab rewards prediction before inspection rather than explanations invented after the graph changes.

1Learn the idea

Read

Lab target

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

Read

Implement one ordinary journey

Build the smallest normal path through POST /v1/summaries for an AI document summarization service. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For Postmortems, the path must contribute directly to this goal: turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Run the baseline command:

./ops/postmortem draft INC-274 --from artifacts/timeline.json
./ops/postmortem claims INC-274 --require-evidence

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

Read

Reconcile behavior with telemetry

The acceptance target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. Query container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Read

Establish the baseline envelope

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.

Checking tutor…