Inject a controlled on-call operations failure

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

This page deliberately breaks On-call operations in a disposable or explicitly approved environment. The scenario is inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift and record the effective configuration:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

./ops/game-day provider-errors --rate 0.38 --minutes 20
./ops/notify suppress --target primary --minutes 6 --simulation

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes. The production tradeoff is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.