Eval metrics lab

Mastery: ship checklist

Eval metrics lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minMastery check

Before you start

Why this matters

Read this incident aloud: a canary prompt improves median latency but quietly reduces grounded answers for billing questions. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained.

1Learn the idea

Read

Ship with evidence and rollback controls

Shipping is a decision backed by artifacts, not the moment a command succeeds. The candidate release must show a clean fixture run, aggregate grounded success rate and p95 latency meeting quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0, a security regression run, telemetry from a staging probe, and a named rollback target. Record model, prompt or policy, data, fixture, and price versions so the evidence can be reproduced after dependencies change.

Use a canary or shadow path when live behavior can differ from fixtures. Compare the candidate with the baseline on identical slices. Promotion should be automatic only for reversible, low-risk changes; security boundary changes and outbound actions need human approval. Define rollback triggers before rollout. A dashboard turning red is not enough—write the exact query, evaluation window, minimum sample, and command or alias swap that restores service.

After promotion, watch the short-term burn rate and the slow quality signal. Verify that rag_grounded_success_ratio receives candidate traffic, traces resolve, logs are redacted, and budget limits work. Close the release by linking evidence and assigning follow-up owners. If an incident occurs, stop canary promotion, compare retriever spans, and roll back only if the quality breach persists. The lesson is mastered when another engineer can operate the system from the repository and runbook without oral history.

For mastery, assemble the release evidence and rehearse rollback. A teammate should be able to answer “what changed, what passed, what will page us, and how do we undo it?” from committed artifacts.

Read

Focused implementation artifact

release: candidate
baseline: stable
gates:
  primary_metric: "grounded success rate and p95 latency"
  required: "quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0"
  security_regression: pass
  staging_probe: pass
  telemetry_signal: "rag_grounded_success_ratio"
rollout:
  canary_percent: 5
  rollback_on: "critical failure or sustained SLO breach"
owner: on-call-ai-platform

Read

Rehearse promotion and rollback

Assemble a release record containing candidate and baseline versions, fixture hash, policy and model versions, evaluation report, security result, staging probe trace, budget result, owner, and rollback target. Re-run the known regression an average-latency chart stays green while the p95 and billing groundedness collapse from a clean checkout or equivalent isolated environment. The evidence must show grounded success rate and p95 latency satisfies quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0 without suppressing failed slices.

Send a canary or shadow request and follow its trace from entry to final decision. Confirm rag_grounded_success_ratio receives candidate traffic, redaction remains effective, and the alert points to an actionable runbook. State the rollback trigger with metric, threshold, window, and minimum sample. Then rehearse rollback using a fake alias, feature flag, or action adapter and prove the stable version resumes service.

Promotion requires a named approver for irreversible actions or authority changes. After release, watch fast reliability signals and slower quality/cost signals for the declared period. If the gate or live SLO fails, stop canary promotion, compare retriever spans, and roll back only if the quality breach persists. Close only when another engineer can reproduce the evidence and execute rollback without oral instructions.

Checking tutor…

Continue learning · glossary & guides

Can the release evidence be reproduced independently?
Are promotion and rollback criteria executable?
Do canary telemetry, ownership, and post-release checks work?
Local references: Glossary: SLO · Glossary: observability · Cheatsheet: production ops signals
Previous