Secure incident response operations

You own a citation-producing RAG API at POST /v1/research. The goal is to coordinate detection, severity, roles, mitigation, evidence, and communication during a user-visible correctness incident. The measurable target is declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. The known production tension is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Incident response for a citation-producing RAG API. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

severity_rules:
  sev1:
    any:
      - citation_failure_rate: "> 0.05 for 5m"
      - regulated_tenant_affected: true
roles: [incident_commander, operations, communications, scribe]
mitigations:
  - disable_flag: reranker_v2
  - rollback: reranker
evidence_retention_days: 90

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

sha256sum artifacts/pre-drill/* > artifacts/pre-drill/SHA256SUMS
./ops/evidence redact --input artifacts/pre-drill --policy ops/redaction.yaml

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface citation_validation_failures_total, affected_answers_total{tenant}, incident_mttd_seconds, incident_mttm_seconds, deployment_version, and feature_flag_state must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet declare SEV-1 within 8 minutes when more than 5% of answers cite nonexistent documents, assign commander/comms/operations roles, disable the faulty reranker within 12 minutes, and preserve a timestamped evidence bundle. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario deploy a reranker that reverses relevance scores for one tenant and introduces fabricated citation IDs; responders must distinguish correctness failure from availability and roll back only the implicated component. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was reranker v2.7 sorted ascending scores, producing 11.8% invalid citations for legal-research users while HTTP latency and status dashboards remained green. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is fast rollback limits harm but may erase runtime evidence; broad shutdown is simple but denies service to unaffected tenants that could safely remain online. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.