Canary and operate on-call operations

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

Before shipping On-call operations, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./ops/handoff create --rotation ai-prod --template ops/handoff.md
./ops/oncall report --rotation ai-prod --window 28d

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes.—as a regression scenario. The enduring production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.