Test service-level objectives signals and thresholds

Instrument AI support-answer service with 99.5% availability + TTFT SLO so reliability owner gating releases on error budget can answer: demand, errors, latency/age, saturation. Emit fields needed by slo:availability:ratio and burn_rate1h with bounded labels. Sample successful high-volume traces; keep errors and rollout transitions denser within policy.

Define at least one alert that would fire for SLO-VANITY-200-18, with a for/pending window that survives deploy blips. Missing scrape or missing revision labels must not look like health. Include a trace/log example id format you will actually search.

Add a unit/integration check that metrics increment on the happy path and on the HTTP 200 fallbacks counted as good while citations empty — vanity availability branch. Store machine-readable output in CI artifacts when practical.

Cardinality discipline: tenant and revision are usually enough; raw question text is not a label. Exemplars or trace links beat screenshots alone when debugging SLO-VANITY-200-18. Define who owns alert fatigue review. If you export to a vendor, record retention and access. Synthetic probes should use non-sensitive fixtures and still exercise POST /v1/support/answer. Practice the query you will type at 2am once, while calm.

Document the exact PromQL or log query in the runbook stub for this service. Verify histograms have buckets around your SLO target. Add a canary synthetic check that exercises the oracle path every few minutes in staging. Confirm that PII redaction happens before export. Track build/version as a label on the golden signals. Delete noisy debug logs before they become accidental product dependencies. In this chapter the product is AI support-answer service with 99.5% availability + TTFT SLO, the human stakeholder is reliability owner gating releases on error budget, and the incident id you design against is SLO-VANITY-200-18. Re-state the oracle in your notes — 3% provider timeout for 20m trips fast-burn; monthly budget math matches calc-budget.py — and keep the invariant visible: bad events include malformed upstream + timeout fallbacks; auth 4xx excluded. Track slo:availability:ratio and burn_rate1h as the scoreboard. Surface under change control: POST /v1/support/answer. If you only have forty minutes, finish the fixture for HTTP 200 fallbacks counted as good while citations empty — vanity availability before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.

Set a timer for twelve minutes and attempt only the observability concerns for SLO lab. Speak aloud what reliability owner gating releases on error budget would see on success versus on HTTP 200 fallbacks counted as good while citations empty — vanity availability. Write three bullet artifacts you must keep: a fixture name tied to SLO-VANITY-200-18, a metric query for slo:availability:ratio and burn_rate1h, and the rollback or refuse command for POST /v1/support/answer. Then extend the notes with one sentence on how bad events include malformed upstream + timeout fallbacks; auth 4xx excluded would be violated if you skipped this page. Close by restating the oracle in your own words without looking: 3% provider timeout for 20m trips fast-burn; monthly budget math matches calc-budget.py.