AnyoneLearnAI
Start learning
Chapter DProduction monitoring labPage 2 of 8

Production monitoring lab

Set up interfaces and contracts

Production monitoring lab is production work only when one frozen failure can be reproduced, one measurable gate can stop a release, and one operator can safely reverse it.

~14 minSetup

Before you start

Why this matters

Read this incident aloud: retrieval returns fewer chunks after an index refresh while HTTP status and model latency remain normal. In two minutes, write the earliest deterministic check that should fail, the telemetry signal you would inspect, and the action that must not happen automatically. Compare your answer with this chapter's boundary: operators see aggregates and redacted traces; raw user text requires break-glass access.

1Learn the idea

Read

Build typed boundaries and fixtures

Code begins at the boundary, not at the provider SDK. Parse TelemetryEvent with release, request_id, intent, retrieved_count, groundedness, latency_ms, status, and cost_usd before any model or tool call, and return IncidentDecision with severity, violated_slos, suspected_span, action, and rollback_release even when the provider times out. Typed records make malformed data a normal error path instead of an exception discovered after a side effect. Inject the model client, clock, action adapter, and telemetry sink so tests can replace each one. That design also prevents a local test from accidentally reaching production.

The important implementation choice is ordering. Authenticate and normalize first; make the controlled call second; validate the response third; commit any state or action last. For this system, operators see aggregates and redacted traces; raw user text requires break-glass access. A convenient function that mixes parsing, generation, action, and logging cannot prove that ordering. Keep each stage named so a trace can show which boundary rejected the request.

Use deterministic identifiers derived from stable business inputs when retries are possible. Record the release, fixture version, and policy version with the result. Never derive authorization from generated text. The code path should make the safe behavior easier than bypassing it: callers receive a decision object, not an unrestricted provider response.

For setup, define dataclasses or schemas, dependency protocols, and a fixture loader. Validate fixture uniqueness at startup and fail closed if required policy fields are absent. The contract should make illegal states difficult to construct.

Read

Focused implementation artifact

from dataclasses import dataclass

@dataclass(frozen=True)
class LabInput:
    payload: dict
    release: str
    trace_id: str

@dataclass(frozen=True)
class LabResult:
    passed: bool
    reasons: tuple[str, ...]
    metrics: dict[str, float]

def load_fixture() -> LabInput:
    payload = {"release":"2026.07.18-canary","request_id":"req_a19","intent":"account_recovery","retrieved_count":0,"groundedness":0.22,"latency_ms":1180,"status":200,"cost_usd":0.011}
    assert payload, "fixture must not be empty"
    return LabInput(payload, "candidate", "trace-test-001")

Read

Exercise the contract

Construct the fixture through the typed input, then deliberately remove one required field and add one unknown field. Decide whether each is rejected at parse time or represented explicitly. A permissive dictionary that silently drops data is not a contract. Record validation errors with a fixture ID and trace ID, but not the rejected payload.

Next, substitute fake adapters for model, storage, clock, telemetry, and external actions. Each fake should record calls, support a deterministic response, and default to denying network or side effects. Use those records to prove that a malformed input causes zero provider calls. The contract's primary result should carry enough data to compute grounded answer rate by intent and release and emit support_answer_grounded_ratio without reopening raw input.

Recreate the global dashboard masks a 35% quality drop limited to account-recovery requests as a fixture-construction test. If the type system cannot express the expectation, add an explicit policy field rather than hiding it in a comment. Keep the boundary enforceable in code. On violation, freeze promotion, page the owner, inspect retrieval traces, then roll back the index alias if confirmed.

Checking tutor…

Continue learning · glossary & guides