Canary and operate disaster recovery

You own a regional RAG platform with PostgreSQL, object storage, and vector indexes at primary us-east-1 and recovery us-west-2. The goal is to prove recovery objectives through restore and failover, including data integrity and application correctness rather than infrastructure availability alone. The measurable target is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. The known production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora.

Before shipping Disaster recovery, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is meet a 15-minute recovery point objective and 60-minute recovery time objective; restore the latest consistent PostgreSQL backup, rebuild vector indexes from immutable documents, route 10% then 100% traffic, and keep citation mismatch below 0.1%. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

objectives:
  rpo_minutes: 15
  rto_minutes: 60
recovery_order: [postgres, documents, vector_index, api, dns]
validation:
  database_checksum: required
  citation_mismatch_ratio_max: 0.001
traffic_steps_percent: [10, 50, 100]
abort_on: [checksum_failure, mismatch_budget_exceeded]

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./dr traffic set --region us-west-2 --percent 10
./dr traffic promote --steps 50,100 --observe-minutes 10

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series backup_age_seconds, backup_restore_duration_seconds, vector_rebuild_progress_ratio, dns_failover_seconds, citation_mismatch_ratio, and recovery_checkpoint_timestamp. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: simulate total loss of us-east-1 plus a corrupted latest vector snapshot; select the prior verified database backup, rebuild embeddings, and document the exact data-loss boundary before traffic cutover. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—a regional control-plane outage coincided with a silently corrupt vector snapshot; DNS moved in 11 minutes but answers cited stale documents because application validation was skipped.—as a regression scenario. The enduring production tension is hot standby minimizes RTO but doubles substantial model and data cost; rebuildable vector indexes reduce backup complexity but can dominate recovery time for large corpora. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.