Canary and operate incident response

You own a citation-producing RAG API at POST /v1/research. The goal is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. The measurable target is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. The known production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online.

Before shipping Incident response, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

severity_rules:
  sev1:
    any:
      - citation_failure_rate: "> 0.05 for 5m"
      - regulated_tenant_affected: true
roles: [incident_commander, operations, communications, scribe]
mitigations:
  - disable_flag: reranker_v2
  - rollback: reranker
evidence_retention_days: 90

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./ops/incident close SIM-CITATIONS --verify-metric citation_failure_rate<0.005
./ops/review schedule SIM-CITATIONS --within-hours 48

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series citation_validation_failures_total, affected_answers_total{tenant}, incident_mttd_seconds, incident_mttm_seconds, deployment_version, and feature_flag_state. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: deploy a reranker that reverses relevance scores for one tenant and introduces fabricated citation IDs; responders must distinguish correctness failure from availability and roll back only the implicated component. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—reranker v2.7 sorted ascending scores, producing 11.8% invalid citations for legal-research users while HTTP latency and status dashboards remained green.—as a regression scenario. The enduring production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.