Secure service-level objectives operations

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Service-level objectives for an AI support-answer service. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

opa eval -d policy/release.rego -i artifacts/budget.json 'data.release.allow'
./scripts/audit-slo-exclusions.sh

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.