Triage and recover service-level objectives

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

Assume the bounded failure is active: replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip. Begin with the user-visible objective—99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

amtool alert query SupportAnswerFastBurn
curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[]|{labels,state}'

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.