Secure postmortems operations

You own an AI document summarization service at POST /v1/summaries. The goal is to turn incident evidence into a blameless causal analysis with owned corrective actions that reduce recurrence and detection time. The measurable target is publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. The known production tension is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Postmortems for an AI document summarization service. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

postmortem:
  incident_id: INC-274
  impact_window_utc: [14:03, 14:20]
  sections: [impact, detection, timeline, causal_analysis, response, actions]
  actions_require: [owner, due_date, verifier, evidence]
  prohibited: [blame, counterfactual_certainty, unattributed_quotes]
review_due_business_days: 5

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

./ops/postmortem redact INC-274 --policy ops/privacy.yaml
./ops/actions audit INC-274 --require owner,due_date,verifier,evidence

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface container_oom_events_total, prompt_tokens, deployment_version, unavailable_requests_total, incident_action_overdue_total, and recurrence_test_passed must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet publish within five business days, reconcile logs and deployment events to one UTC timeline, separate trigger from contributing conditions, and assign every action an owner, due date, verification method, and risk-reduction claim. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario analyze a supplied incident where a tokenizer upgrade doubled prompt size, exhausted worker memory, and caused 17 minutes of partial outage; reject hindsight claims unsupported by timestamped evidence. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was tokenizer v4 counted whitespace differently, doubled p95 prompt tokens, OOM-killed workers, and returned 503 for 23% of summary requests from 14:03 to 14:20 UTC. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is deep analysis finds systemic conditions but delays publication; many corrective actions feel thorough but dilute ownership and hide which controls materially reduce risk. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.