AnyoneLearnAI
Start learning
Chapter DEval metrics labPage 4 of 8

Eval metrics lab

Validate outputs and schemas

Eval metrics lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minValidation

Before you start

Why this matters

Read this incident aloud: a canary prompt improves median latency but quietly reduces grounded answers for billing questions. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained.

1Learn the idea

Read

Test behavior and evaluate quality

Testing this system requires more than checking that JSON parses. Build a small matrix with one normal case, one boundary case, the known failure, and one adversarial case. Freeze randomness and time. Stub the provider with recorded semantic outcomes rather than brittle prose snapshots. Then assert the decision, prohibited behavior, emitted metric, and absence of sensitive fields. The key behavioral assertion for this topic is assert report.p95_ms < 3000 and report.quality_rate >= 0.92.

Evaluate at two resolutions. First, case-level reasons must tell a developer exactly which expectation failed. Second, aggregate grounded success rate and p95 latency must meet quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0 on the frozen set and on important slices. A global pass can hide a severe intent, carrier, release, or customer workflow. Report numerator and denominator beside every rate; 0 failures over two cases is not strong evidence.

Before accepting a metric, attack it. If a shorter refusal raises a keyword score while usefulness collapses, the metric is being gamed. If lower cost excludes retries, the denominator is wrong. If latency ignores timeouts, the sample is censored. Compare automated scores with a small blinded human review and record disagreements as future fixtures.

For validation, execute the full matrix and print a machine-readable report plus a short developer summary. Exit nonzero on a violated critical gate; do not round a failing value into a pass.

Read

Focused implementation artifact

import pytest

CASES = [
    pytest.param({"route":"/api/chat","status":200,"latency_ms":4210,"groundedness":0.61,"tokens":1840,"cost_usd":0.018,"release":"canary-v17"}, id="known-boundary"),
]

@pytest.mark.parametrize("case", CASES)
def test_behavioral_gate(case):
    report = aggregate(samples, window="5m", group_by=["release", "intent"])
    assert report.p95_ms < 3000 and report.quality_rate >= 0.92

def test_release_gate(report):
    assert report.sample_count >= 10
    assert report.critical_failures == 0

Read

Build the evaluation report

Expand the parameterized test into four named fixtures: benign control, threshold boundary, known regression, and adversarial misuse. For every row, assert the decision and at least one negative fact, such as no leak, no unauthorized action, no duplicate write, or no candidate promotion. Provider prose can vary; policy outcomes and required evidence cannot.

Aggregate the case results into grounded success rate and p95 latency and preserve numerator, denominator, critical failures, and slice keys. Apply quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0 exactly. Compare baseline and candidate on the same fixtures, then review disagreements where the automated score and a blinded human label diverge. Those disagreements are useful data, not noise to discard.

The regression named an average-latency chart stays green while the p95 and billing groundedness collapse must fail before the fix and pass after it. Verify rag_grounded_success_ratio is emitted for both outcomes and that the failing report points to fixture IDs and trace IDs without embedding sensitive content. A failed gate leads operators to stop canary promotion, compare retriever spans, and roll back only if the quality breach persists.

Checking tutor…

Continue learning · glossary & guides