Define the lab goal and success criteria

Start by drawing the boundary around an SLO dashboard and alert evaluator for a production RAG endpoint. The unit under control is not “the model”; it is one repeatable transaction from RequestSample with route, status, latency_ms, groundedness, tokens, cost_usd, and release to WindowReport with p50, p95, error_rate, quality_rate, cost_per_success, and burn_rate. In this lab, a canary prompt improves median latency but quietly reduces grounded answers for billing questions. That makes the learning target observable: we can replay the transaction, record the same fields, and decide whether a release is safer than its baseline. The primary decision metric is grounded success rate and p95 latency. The release rule is quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0. Those values are deliberately paired: a single average cannot hide a critical subgroup or a security escape.

Write the failure before writing implementation code. Our named reproduction is: an average-latency chart stays green while the p95 and billing groundedness collapse. A useful reproduction contains a frozen input, an expected decision, the release identifier, and enough deterministic dependencies to rerun locally. It must not depend on a live customer, changing clock, or unrestricted provider account. The fixture below is synthetic but shaped like production. Keep it in version control and add a case whenever an incident exposes a missing behavior.

Success has three levels. A request-level assertion explains one example. An aggregate metric catches drift over many examples. An operational rule says who acts and how quickly. Here, rag_grounded_success_ratio records the aggregate, while the request artifact carries a correlation ID. The owner should be able to move from an alert to the exact failing fixture without searching raw prompts. The security boundary is equally explicit: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained.

The deliverable is a one-page acceptance contract plus the first red test. Do not tune a model yet; uncertainty about the contract is the bug to remove on this page.

Use the fixture above as the first row of an acceptance table. Add columns for expected decision, prohibited side effect, severity, policy version, and evidence to retain. The failure statement—an average-latency chart stays green while the p95 and billing groundedness collapse—must be falsifiable: a teammate should know exactly which assertion turns red. Write the baseline result before trying a candidate so improvement is comparative rather than anecdotal.

Now calculate grounded success rate and p95 latency by hand for a three-case toy set: one pass, one ordinary failure, and one critical failure. Apply the published gate, quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0, without rounding. This exposes whether weighting, critical-case overrides, and empty slices behave as intended. Emit rag_grounded_success_ratio once for the toy run and verify that its labels contain a bounded release and outcome, never fixture text.

The lab is ready only when its fixture is synthetic, its dangerous actions are replaced by recording fakes, and its evidence can be retained safely. The expected incident response is to stop canary promotion, compare retriever spans, and roll back only if the quality breach persists. Assign that decision to a role, not “the team,” and state which artifact proves recovery.