AnyoneLearnAI
Start learning
Chapter DEval metrics labPage 6 of 8

Eval metrics lab

Add observability and tests

Eval metrics lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minObservability

Before you start

Why this matters

Read this incident aloud: a canary prompt improves median latency but quietly reduces grounded answers for billing questions. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: telemetry excludes prompt text and customer identifiers; only pseudonymous request IDs are retained.

1Learn the idea

Read

Instrument metrics, logs, and traces

Instrumentation should answer four questions from one correlation ID: what release ran, which stage failed, how long each stage took, and what decision was made. Emit structured events, not interpolated prose. A useful event includes event, request_id, trace_id, release, stage, outcome, latency_ms, and a small set of bounded labels. Record rag_grounded_success_ratio as a counter or ratio with stable dimensions; never use request IDs, user text, or error messages as metric labels.

Trace the transaction using a root span and child spans around meaningful boundaries. Add counts, versions, finish reasons, and safe IDs. Do not attach full prompts, retrieved passages, secrets, or tool arguments. Logs explain discrete decisions, metrics detect population changes, and traces reconstruct one request; none replaces the others. Sampling may reduce ordinary success traces, but always retain errors and policy denials under the approved retention policy.

Create an operational test that sends a synthetic request tagged probe=true, then checks that the metric increments, the trace has required child spans, and the log can be found by trace ID. An alert is useful only if it links to a runbook and has a clear action. For this lab, the release rule remains quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0, and the first response is to stop canary promotion, compare retriever spans, and roll back only if the quality breach persists.

For observability, test telemetry as a product interface. Assert required span names and log keys, cardinality-safe dimensions, redaction, and alert routing using synthetic events.

Read

Focused implementation artifact

def emit_decision(metrics, logger, *, trace_id, release, outcome, latency_ms):
    metrics.counter("rag_grounded_success_ratio").add(1, {"release": release, "outcome": outcome})
    logger.info("canary_prompt_v17", extra={
        "trace_id": trace_id, "release": release,
        "outcome": outcome, "latency_ms": latency_ms,
        "payload": "[REDACTED]",
    })

def test_telemetry(fake_metrics, fake_logger):
    emit_decision(fake_metrics, fake_logger, trace_id="t-1", release="canary", outcome="deny", latency_ms=18)
    assert fake_logger.last["payload"] == "[REDACTED]"

Read

Verify the telemetry path

Create a synthetic request that reaches every normal stage and another that reproduces an average-latency chart stays green while the p95 and billing groundedness collapse. The success trace must contain a root span and ordered child spans; the failure trace must mark the first broken boundary and skip spans for actions that never ran. Assert context propagation rather than relying on visually similar timestamps.

Emit rag_grounded_success_ratio with release, outcome, and a bounded domain slice. Keep request IDs in logs and traces, never metric labels. A structured log should include trace ID, stage, decision, latency, and policy or model version. Redaction happens before export and is tested against nested secrets and identifiers. Sampling may drop ordinary successes, but it must retain errors and policy denials under the documented retention cap.

Test the alert using synthetic events until it opens the runbook with the correct query. The alert condition derives from grounded success rate and p95 latency and quality at least 0.92, p95 below 3000 ms, and burn rate below 2.0; include window and minimum sample so one request cannot page an entire team. The first response is to stop canary promotion, compare retriever spans, and roll back only if the quality breach persists.

Checking tutor…

Continue learning · glossary & guides