Secure on-call operations operations

You own a production AI answer platform at PagerDuty service ai-answer-prod. The goal is to build a humane rotation and a first-15-minutes response that converts actionable symptoms into ownership, mitigation, and handoff. The measurable target is SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. The known production tension is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back On-call operations for a production AI answer platform. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

service: ai-answer-prod
escalation:
  - target: primary
    timeout_minutes: 5
  - target: secondary
    timeout_minutes: 5
  - target: incident_commander
    timeout_minutes: 5
severities:
  sev1: page
  sev2: page_business_hours
  sev3: ticket

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

./ops/access audit --group ai-prod-oncall
./ops/access verify-breakglass --expires-within 60m

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface page_delivery_seconds, page_ack_seconds, incidents_total{severity}, alert_actionability_ratio, handoff_age_hours, and responder pages per shift must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet SEV-1 pages acknowledge within 5 minutes, an incident commander is named within 10 minutes, mitigation begins within 15 minutes, and fewer than two non-actionable pages reach a person per shift. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario inject elevated provider 503s at 02:00 in the simulation, suppress the primary notification, and verify escalation reaches secondary in 5 minutes while the responder uses a bounded fallback rather than restarting everything. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was provider 503s rose to 38%; the primary missed a phone push, the secondary restarted healthy pods, and absence of an incident commander delayed fallback activation by 22 minutes. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is aggressive paging shortens detection but creates fatigue and attrition; wide access speeds mitigation but increases the blast radius of a tired responder. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.