Inject a controlled disaster recovery failure

You own a regional RAG platform with PostgreSQL, object storage, and vector indexes at primary us-east-1 and recovery us-west-2. The goal is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. The measurable target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. The known production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora.

This page deliberately breaks Disaster recovery in a disposable or explicitly approved environment. The scenario is simulate total loss of us-east-1 plus a corrupted latest vector snapshot; select the prior verified database backup, rebuild embeddings, and document the exact data-loss boundary before traffic cutover. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of backup_age_seconds, backup_restore_duration_seconds, vector_rebuild_progress_ratio, dns_failover_seconds, citation_mismatch_ratio, and recovery_checkpoint_timestamp and record the effective configuration:

objectives:
  rpo_minutes: 15
  rto_minutes: 60
recovery_order: [postgres, documents, vector_index, api, dns]
validation:
  database_checksum: required
  citation_mismatch_ratio_max: 0.001
traffic_steps_percent: [10, 50, 100]
abort_on: [checksum_failure, mismatch_budget_exceeded]

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

./dr game-day --lose-region us-east-1 --corrupt-latest-vector-snapshot
./dr rebuild-vectors --source s3://documents-immutable --checkpoint 10000

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: a regional control-plane outage coincided with a silently corrupt vector snapshot; DNS moved in 11 minutes but answers cited stale documents because application validation was skipped. The production tradeoff is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.