Test service-level objectives signals and thresholds

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

Convert the Service-level objectives objective into tests, not screenshots. The target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. Write one assertion for the exact boundary, one immediately inside it, and one immediately outside it. Include denominator behavior when traffic is zero, missing, duplicated, delayed, or partially sampled.

The telemetry contract is sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps. For every series, verify metric type, unit, label allowlist, ownership, and retention. Counters must only rise; histograms need buckets around the target; gauges require staleness handling. Confirm dashboards display demand, successful or protected outcomes, user latency or age, saturation, and version context together. A green ratio without its event count is not sufficient evidence.

Run the test workflow:

promtool test rules monitoring/support-answer-slo.test.yaml
pytest -q tests/test_sli_classification.py

Store machine-readable output as a build artifact. Seed deterministic time or fixtures where possible, but keep at least one integration test against the actual collector and rule engine. Tests must fail when the threshold expression, label name, or alert receiver is intentionally broken. A test that remains green after its signal is disconnected is worse than no test because it creates false confidence.

Use the configured contract:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Replay the controlled scenario—replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip.—and measure time from fault start to metric visibility, alert pending, alert firing, delivery, acknowledgement where relevant, mitigation, and stable recovery. Compare those times with the operational objective. Verify short benign blips do not page if the policy says they should not, while sustained or fast-burning impact does.

Check telemetry loss separately from service loss. Stop or misconfigure one scrape target and prove the system reports missing data rather than silently treating it as healthy. Verify logs and drill artifacts redact secrets and private payloads. Bound label cardinality using a representative workload and set an alert or budget for unexpected series growth.

Use the prior incident—the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents.—to ask which test would have detected the problem earlier. Remove or demote signals that do not lead to a decision. The production tradeoff is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. Quantify the cost of collection and paging against the risk reduction it buys.

This page passes only when contract tests, failure tests, alert-path tests, and recovery assertions all run from a clean environment and produce retained evidence linked to the configuration revision.