Inject a controlled service-level objectives failure

You own an AI support-answer service at POST /v1/support/answer. The goal is to turn user-visible availability and latency into measurable SLIs, an error budget, and release decisions that resist vanity averages. The measurable target is 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. The known production tension is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget.

This page deliberately breaks Service-level objectives in a disposable or explicitly approved environment. The scenario is replay a 3% provider timeout rate for 20 minutes and prove the 1-hour/5-minute burn-rate alert fires before the monthly budget is exhausted without paging on a two-minute deploy blip. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of sli_events_total{indicator,outcome}, slo_error_budget_remaining_ratio, slo_burn_rate, time_to_first_token_seconds, and alert delivery timestamps and record the effective configuration:

groups:
- name: support-answer-slo
  rules:
  - record: slo:availability:ratio_rate5m
    expr: sum(rate(sli_events_total{indicator="availability",outcome="good"}[5m]))
      / sum(rate(sli_events_total{indicator="availability"}[5m]))
  - alert: SupportAnswerFastBurn
    expr: slo:availability:burn_rate1h > 14.4 and slo:availability:burn_rate5m > 14.4
    for: 2m

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

toxiproxy-cli toxic add provider -t timeout -a timeout=3000
vegeta attack -rate=30/s -duration=20m < fixtures/targets.txt | vegeta report

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains 99.5% good-event availability over 28 days and 95% of eligible answers with time-to-first-token below 1.2 seconds; exclude authenticated client 4xx but count malformed upstream output and timeout fallbacks as bad events. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: the dashboard showed 99.9% HTTP success because fallback responses were 200, while 7% contained empty citations and were unusable to support agents. The production tradeoff is a strict SLO protects users but can freeze valuable releases; broad eligibility makes gaming harder but may charge product experiments against the reliability budget. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.