Run the on-call operations baseline

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

Build the smallest normal path through PagerDuty service ai-answer-prod for a production AI answer platform. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For On-call operations, the path must contribute directly to this goal: build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Run the baseline command:

./ops/incident declare --severity 2 --title 'provider 503 simulation' --dry-run
./ops/status snapshot --service ai-answer-prod

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

The acceptance target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. Query page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.