Triage and recover on-call operations

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

Assume the bounded failure is active: inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything. Begin with the user-visible objective—SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

./ops/timeline export --incident SIM-2026-07-18
./ops/metrics query 'page_ack_seconds{service="ai-answer-prod"}' --since 30m

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.