Define the production target for disaster recovery

You own a regional RAG platform with PostgreSQL, object storage, and vector indexes at primary us-east-1 and recovery us-west-2. The goal is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. The measurable target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. The known production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora.

Start with the request or operator journey, not a tool choice. For Disaster recovery, draw primary us-east-1 and recovery us-west-2 entering a regional RAG platform with PostgreSQL, object storage, and vector indexes, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: a regional control-plane outage coincided with a silently corrupt vector snapshot; DNS moved in 11 minutes but answers cited stale documents because application validation was skipped. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is simulate total loss of us-east-1 plus a corrupted latest vector snapshot; select the prior verified database backup, rebuild embeddings, and document the exact data-loss boundary before traffic cutover. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are backup_age_seconds, backup_restore_duration_seconds, vector_rebuild_progress_ratio, dns_failover_seconds, citation_mismatch_ratio, and recovery_checkpoint_timestamp. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

objectives:
  rpo_minutes: 15
  rto_minutes: 60
recovery_order: [postgres, documents, vector_index, api, dns]
validation:
  database_checksum: required
  citation_mismatch_ratio_max: 0.001
traffic_steps_percent: [10, 50, 100]
abort_on: [checksum_failure, mismatch_budget_exceeded]

Review every number against demand and consequence. hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

./dr inventory --region us-west-2
./dr backups list --verified --before now --format table

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.