Instrument metrics and rollout decisions

Emit one structured event at each control point: request accepted, policy or variant selected, dependency completed, result produced, and rollback or promotion changed. Every event needs timestamp, request ID, release revision, configuration revision, outcome category, and duration. Add only low-cardinality dimensions to metrics; put request IDs in traces and logs, not metric labels.

The primary dashboard reports authentication failures, consumers by credential version, secret-read anomalies, and time from issue to old-key revocation. Show numerator and denominator, cohort or revision splits, sample count, and a comparable control. Add saturation and dependency health so operators can tell a bad release from a shared outage. Quality metrics need dataset or grader revision next to the score.

Page on symptoms that require immediate action: sustained availability loss, rapid error-budget burn, security rejection anomaly, or queue age threatening the stated objective. Create tickets for slow trends. Every alert links to a runbook and includes the affected revision or cohort. Test alert queries against a synthetic event before trusting them.

Avoid averages that erase harm. Compare p50 and p95, inspect tenant or cohort slices with privacy-safe cardinality, and display absolute counts beside rates. Low traffic can make a zero-error canary meaningless. The dashboard must visibly mark insufficient data and stale series; rollout automation must treat both as hold.

Follow this policy: never print secret values, use workload identity to read one named secret, separate issuer and deployer roles, and retain immutable audit events. Redact at collection, not only in the dashboard. Sample successful high-volume traces, but retain errors and rollout transitions at a higher rate within policy. Restrict access to sensitive operational data and set retention intentionally.

Run a trace exercise using one fixture. Starting from its public request ID, locate the selected policy, external calls, final disposition, and emitted metric within five minutes. Then start from a breached chart and find representative traces. Both directions matter during an incident.

Populate the gate function from the validation chapter using a fixed observation window. Require minimum traffic and complete guardrails. Record the output and reasons as a deployment event. A human release commander reviews quality examples before promotion even when numeric gates pass. The completed observability package includes dashboard definition, alert tests, trace evidence, and a decision record.

Define the alert as versioned configuration, including a minimum window and explicit no-data behavior. Labels remain low cardinality; request_id belongs in traces, not metric labels.

alert: secrets-rotation-lab-secret_consumers_by_version
expr: secret_consumers_by_version > 1
for: 5m
labels:
  severity: page
  owner: building-labs
annotations:
  action: hold rollout and inspect revision
  no_data: hold

A representative event connects the chart to one decision without prompts, secret bytes, or raw customer identifiers. The local synthetic probe should emit this shape:

{
  "event": "release_guardrail_evaluated",
  "revision": "provider-key-v2",
  "config_revision": "local-fixture-1",
  "outcome": "hold",
  "reason": "secret_consumers_by_version_threshold",
  "sample_count": 1200,
  "duration_ms": 37
}

Validate with one below-threshold and one above-threshold sample. Expected behavior is inactive first and firing only after five continuous minutes second; missing samples must hold. If events exist but the graph is empty, compare metric names and labels before changing the query. If no trace can be found, verify identity propagation and ensure it was not made a high-cardinality metric label.