Triage and recover incident response

You own a citation-producing RAG API at POST /v1/research. The goal is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. The measurable target is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. The known production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online.

Assume the bounded failure is active: deploy a reranker that reverses relevance scores for one tenant and introduces fabricated citation IDs; responders must distinguish correctness failure from availability and roll back only the implicated component. Begin with the user-visible objective—declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query citation_validation_failures_total, affected_answers_total{tenant}, incident_mttd_seconds, incident_mttm_seconds, deployment_version, and feature_flag_state by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

./ops/timeline follow SIM-CITATIONS
./ops/logs query 'citation_valid=false' --since 30m --group-by version

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

severity_rules:
  sev1:
    any:
      - citation_failure_rate: "> 0.05 for 5m"
      - regulated_tenant_affected: true
roles: [incident_commander, operations, communications, scribe]
mitigations:
  - disable_flag: reranker_v2
  - rollback: reranker
evidence_retention_days: 90

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was reranker v2.7 sorted ascending scores, producing 11.8% invalid citations for legal-research users while HTTP latency and status dashboards remained green. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.