Triage and recover disaster recovery

You own a regional RAG platform with PostgreSQL, object storage, and vector indexes at primary us-east-1 and recovery us-west-2. The goal is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. The measurable target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. The known production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora.

Assume the bounded failure is active: simulate total loss of us-east-1 plus a corrupted latest vector snapshot; select the prior verified database backup, rebuild embeddings, and document the exact data-loss boundary before traffic cutover. Begin with the user-visible objective—meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query backup_age_seconds, backup_restore_duration_seconds, vector_rebuild_progress_ratio, dns_failover_seconds, citation_mismatch_ratio, and recovery_checkpoint_timestamp by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

./dr status --watch --interval 30s
./dr compare --primary-checkpoint "$LAST_GOOD" --recovery-checkpoint "$RESTORED"

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

objectives:
  rpo_minutes: 15
  rto_minutes: 60
recovery_order: [postgres, documents, vector_index, api, dns]
validation:
  database_checksum: required
  citation_mismatch_ratio_max: 0.001
traffic_steps_percent: [10, 50, 100]
abort_on: [checksum_failure, mismatch_budget_exceeded]

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was a regional control-plane outage coincided with a silently corrupt vector snapshot; DNS moved in 11 minutes but answers cited stale documents because application validation was skipped. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.