Define the production target for on-call operations

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

Start with the request or operator journey, not a tool choice. For On-call operations, draw PagerDuty service ai-answer-prod entering a production AI answer platform, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Review every number against demand and consequence. aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

./ops/oncall lint ops/oncall.yaml
./ops/oncall show --at '2026-07-18T02:00:00Z' --timezone UTC

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.