Instrument metrics and rollout decisions

Emit one structured event at each control point: request accepted, policy or variant selected, dependency completed, result produced, and rollback or promotion changed. Every event needs timestamp, request ID, release revision, configuration revision, outcome category, and duration. Add only low-cardinality dimensions to metrics; put request IDs in traces and logs, not metric labels.

The primary dashboard reports revision-sliced error rate, p95 latency, saturation, and grounded-answer rate. Show numerator and denominator, cohort or revision splits, sample count, and a comparable control. Add saturation and dependency health so operators can tell a bad release from a shared outage. Quality metrics need dataset or grader revision next to the score.

Page on symptoms that require immediate action: sustained availability loss, rapid error-budget burn, security rejection anomaly, or queue age threatening the stated objective. Create tickets for slow trends. Every alert links to a runbook and includes the affected revision or cohort. Test alert queries against a synthetic event before trusting them.

Avoid averages that erase harm. Compare p50 and p95, inspect tenant or cohort slices with privacy-safe cardinality, and display absolute counts beside rates. Low traffic can make a zero-error canary meaningless. The dashboard must visibly mark insufficient data and stale series; rollout automation must treat both as hold.

Follow this policy: pin image digests, verify build provenance, use a canary service account with least privilege, and prevent debug headers from exposing routing internals. Redact at collection, not only in the dashboard. Sample successful high-volume traces, but retain errors and rollout transitions at a higher rate within policy. Restrict access to sensitive operational data and set retention intentionally.

Run a trace exercise using one fixture. Starting from its public request ID, locate the selected policy, external calls, final disposition, and emitted metric within five minutes. Then start from a breached chart and find representative traces. Both directions matter during an incident.

Populate the gate function from the validation chapter using a fixed observation window. Require minimum traffic and complete guardrails. Record the output and reasons as a deployment event. A human release commander reviews quality examples before promotion even when numeric gates pass. The completed observability package includes dashboard definition, alert tests, trace evidence, and a decision record.

Define the alert as versioned configuration, including a minimum window and explicit no-data behavior. Labels remain low cardinality; request_id belongs in traces, not metric labels.

alert: canary-deploy-lab-answer_errors_by_revision
expr: answer_errors_by_revision > 0.01
for: 5m
labels:
  severity: page
  owner: building-labs
annotations:
  action: hold rollout and inspect revision
  no_data: hold

A representative event connects the chart to one decision without prompts, secret bytes, or raw customer identifiers. The local synthetic probe should emit this shape:

{
  "event": "release_guardrail_evaluated",
  "revision": "answer-api@sha256:9e1c",
  "config_revision": "local-fixture-1",
  "outcome": "hold",
  "reason": "answer_errors_by_revision_threshold",
  "sample_count": 1200,
  "duration_ms": 37
}

Validate with one below-threshold and one above-threshold sample. Expected behavior is inactive first and firing only after five continuous minutes second; missing samples must hold. If events exist but the graph is empty, compare metric names and labels before changing the query. If no trace can be found, verify identity propagation and ensure it was not made a high-cardinality metric label.