Canary and operate service-level objectives

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

Before shipping Service-level objectives, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./scripts/release-gate.sh --service support-answer --version "$GIT_SHA"
./scripts/slo-report.sh --window 28d

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents.—as a regression scenario. The enduring production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.