Run the disaster recovery baseline

You own a regional RAG platform with PostgreSQL, object storage, and vector indexes at primary us-east-1 and recovery us-west-2. The goal is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. The measurable target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. The known production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora.

Build the smallest normal path through primary us-east-1 and recovery us-west-2 for a regional RAG platform with PostgreSQL, object storage, and vector indexes. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For Disaster recovery, the path must contribute directly to this goal: prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

objectives:
  rpo_minutes: 15
  rto_minutes: 60
recovery_order: [postgres, documents, vector_index, api, dns]
validation:
  database_checksum: required
  citation_mismatch_ratio_max: 0.001
traffic_steps_percent: [10, 50, 100]
abort_on: [checksum_failure, mismatch_budget_exceeded]

Run the baseline command:

./dr restore --backup latest-verified --region us-west-2 --dry-run
./dr validate --checks database-checksum,object-count,citation-sample

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

The acceptance target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. Query backup_age_seconds, backup_restore_duration_seconds, vector_rebuild_progress_ratio, dns_failover_seconds, citation_mismatch_ratio, and recovery_checkpoint_timestamp immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: a regional control-plane outage coincided with a silently corrupt vector snapshot; DNS moved in 11 minutes but answers cited stale documents because application validation was skipped. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.