Inject a controlled incident response failure

You own a citation-producing RAG API at POST /v1/research. The goal is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. The measurable target is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. The known production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online.

This page deliberately breaks Incident response in a disposable or explicitly approved environment. The scenario is deploy a reranker that reverses relevance scores for one tenant and introduces fabricated citation IDs; responders must distinguish correctness failure from availability and roll back only the implicated component. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of citation_validation_failures_total, affected_answers_total{tenant}, incident_mttd_seconds, incident_mttm_seconds, deployment_version, and feature_flag_state and record the effective configuration:

severity_rules:
  sev1:
    any:
      - citation_failure_rate: "> 0.05 for 5m"
      - regulated_tenant_affected: true
roles: [incident_commander, operations, communications, scribe]
mitigations:
  - disable_flag: reranker_v2
  - rollback: reranker
evidence_retention_days: 90

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

./ops/game-day bad-reranker --tenant legal --failure-rate 0.12
./ops/flags set reranker_v2=false --tenant legal --simulation

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: reranker v2.7 sorted ascending scores, producing 11.8% invalid citations for legal-research users while HTTP latency and status dashboards remained green. The production tradeoff is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.