Define the production target for incident response

You own a citation-producing RAG API at POST /v1/research. The goal is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. The measurable target is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. The known production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online.

Start with the request or operator journey, not a tool choice. For Incident response, draw POST /v1/research entering a citation-producing RAG API, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: reranker v2.7 sorted ascending scores, producing 11.8% invalid citations for legal-research users while HTTP latency and status dashboards remained green. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is deploy a reranker that reverses relevance scores for one tenant and introduces fabricated citation IDs; responders must distinguish correctness failure from availability and roll back only the implicated component. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are citation_validation_failures_total, affected_answers_total{tenant}, incident_mttd_seconds, incident_mttm_seconds, deployment_version, and feature_flag_state. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

severity_rules:
  sev1:
    any:
      - citation_failure_rate: "> 0.05 for 5m"
      - regulated_tenant_affected: true
roles: [incident_commander, operations, communications, scribe]
mitigations:
  - disable_flag: reranker_v2
  - rollback: reranker
evidence_retention_days: 90

Review every number against demand and consequence. fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

./ops/incident lint ops/incident-policy.yaml
./ops/incident classify --metric citation_failure_rate=0.118 --tenant legal

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.