Test on-call operations signals and thresholds

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

Convert the On-call operations objective into tests, not screenshots. The target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. Write one assertion for the exact boundary, one immediately inside it, and one immediately outside it. Include denominator behavior when traffic is zero, missing, duplicated, delayed, or partially sampled.

The telemetry contract is page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift. For every series, verify metric type, unit, label allowlist, ownership, and retention. Counters must only rise; histograms need buckets around the target; gauges require staleness handling. Confirm dashboards display demand, successful or protected outcomes, user latency or age, saturation, and version context together. A green ratio without its event count is not sufficient evidence.

Run the test workflow:

pytest -q tests/test_escalation_policy.py
./ops/oncall drill --scenario missed-primary --assert-ack-under 600

Store machine-readable output as a build artifact. Seed deterministic time or fixtures where possible, but keep at least one integration test against the actual collector and rule engine. Tests must fail when the threshold expression, label name, or alert receiver is intentionally broken. A test that remains green after its signal is disconnected is worse than no test because it creates false confidence.

Use the configured contract:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Replay the controlled scenario—inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything.—and measure time from fault start to metric visibility, alert pending, alert firing, delivery, acknowledgement where relevant, mitigation, and stable recovery. Compare those times with the operational objective. Verify short benign blips do not page if the policy says they should not, while sustained or fast-burning impact does.

Check telemetry loss separately from service loss. Stop or misconfigure one scrape target and prove the system reports missing data rather than silently treating it as healthy. Verify logs and drill artifacts redact secrets and private payloads. Bound label cardinality using a representative workload and set an alert or budget for unexpected series growth.

Use the prior incident—provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes.—to ask which test would have detected the problem earlier. Remove or demote signals that do not lead to a decision. The production tradeoff is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. Quantify the cost of collection and paging against the risk reduction it buys.

This page passes only when contract tests, failure tests, alert-path tests, and recovery assertions all run from a clean environment and produce retained evidence linked to the configuration revision.