AnyoneLearnAI
Start learning
Chapter DLLM tracing labPage 3 of 8

LLM tracing lab

Implement the happy path

LLM tracing lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minHappy path

Before you start

Why this matters

Read this incident aloud: a user reports a wrong answer and the trace must distinguish empty retrieval from a generation defect. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: span attributes contain IDs, counts, model versions, and redacted arguments—not prompts, API keys, or PII.

1Learn the idea

Read

Implement the controlled happy path

Code begins at the boundary, not at the provider SDK. Parse TraceContext with request_id, release, hashed_user_id, sampled, and baggage before any model or tool call, and return TraceSummary with root_span, child_spans, durations, safe attributes, status, and error_type even when the provider times out. Typed records make malformed data a normal error path instead of an exception discovered after a side effect. Inject the model client, clock, action adapter, and telemetry sink so tests can replace each one. That design also prevents a local test from accidentally reaching production.

The important implementation choice is ordering. Authenticate and normalize first; make the controlled call second; validate the response third; commit any state or action last. For this system, span attributes contain IDs, counts, model versions, and redacted arguments—not prompts, API keys, or PII. A convenient function that mixes parsing, generation, action, and logging cannot prove that ordering. Keep each stage named so a trace can show which boundary rejected the request.

Use deterministic identifiers derived from stable business inputs when retries are possible. Record the release, fixture version, and policy version with the result. Never derive authorization from generated text. The code path should make the safe behavior easier than bypassing it: callers receive a decision object, not an unrestricted provider response.

For the happy path, run one valid fixture through the real orchestration with fake adapters. Keep the path small: accept, decide, validate, record. A successful response must carry evidence and a trace ID, not merely polished text.

Read

Focused implementation artifact

def run_case(case, *, release, trace_id):
    # Adapters are injected in production; the orchestration owns policy.
    with tracer.start_as_current_span("retrieve") as span: chunks = store.search(query, k=4)
    reasons = evaluate(answer if "answer" in locals() else locals().get("result", locals().get("decision", locals().get("job", locals().get("choice")))))
    return {"passed": not reasons, "reasons": reasons,
            "release": release, "trace_id": trace_id}

result = run_case({"request_id":"req_7b9f2","release":"rag-v12","hashed_user_id":"u_91c","sampled":true,"query":"[REDACTED]"}, release="candidate", trace_id="trace-42")
assert result["trace_id"] == "trace-42"

Read

Walk the successful transaction

Trace one accepted fixture through parse, policy lookup, controlled dependency call, output validation, and result recording. At each stage, name the input and output type. The successful transaction must produce evidence for complete trace ratio and per-span p95 duration, a correlation ID for logs and spans, and a release identifier. It must not grant the model authority to choose identity, credentials, network destinations, or approval.

Run the path with recording fakes and inspect their call order. The assertion is not merely “returned 200”: exactly the expected adapter was called, its arguments were normalized, no forbidden adapter ran, and llm_trace_complete_ratio recorded a bounded outcome. Repeat with the same idempotency key or request identifier where retries are relevant; the observable decision should remain stable.

Finally, swap in the known failure the retrieve span reports zero chunks but an uncorrelated model log makes generation look guilty while leaving orchestration unchanged. The happy path must stop at the correct boundary instead of manufacturing a success. The operator-facing consequence is to mark retrieval as the fault domain and link the trace ID in the incident without copying prompt content.

Checking tutor…

Continue learning · glossary & guides