Define the production target for service-level objectives

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

Start with the request or operator journey, not a tool choice. For Service-level objectives, draw POST /v1/support/answer entering an AI support-answer service, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Review every number against demand and consequence. a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

promtool check rules monitoring/support-answer-slo.yaml
python scripts/calc-budget.py --slo 99.5 --window-days 28

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.