Instrument metrics and rollout decisions

Instrument answer API + indexer sharing provider key via versioned secrets manager so security engineer rotating openai/prod after contractor offboarding can answer: demand, errors, latency/age, saturation. Emit fields needed by secret_version_share and auth_fail_rate with bounded labels. Sample successful high-volume traces; keep errors and rollout transitions denser within policy.

Define at least one alert that would fire for SEC-REVOKE-EARLY-21, with a for/pending window that survives deploy blips. Missing scrape or missing revision labels must not look like health. Include a trace/log example id format you will actually search.

Add a unit/integration check that metrics increment on the happy path and on the revoke v1 while indexer still cached v1 → 401 storm branch. Store machine-readable output in CI artifacts when practical.

Cardinality discipline: tenant and revision are usually enough; raw question text is not a label. Exemplars or trace links beat screenshots alone when debugging SEC-REVOKE-EARLY-21. Define who owns alert fatigue review. If you export to a vendor, record retention and access. Synthetic probes should use non-sensitive fixtures and still exercise secrets manager path openai/prod. Practice the query you will type at 2am once, while calm.

Document the exact PromQL or log query in the runbook stub for this service. Verify histograms have buckets around your SLO target. Add a canary synthetic check that exercises the oracle path every few minutes in staging. Confirm that PII redaction happens before export. Track build/version as a label on the golden signals. Delete noisy debug logs before they become accidental product dependencies. In this chapter the product is answer API + indexer sharing provider key via versioned secrets manager, the human stakeholder is security engineer rotating openai/prod after contractor offboarding, and the incident id you design against is SEC-REVOKE-EARLY-21. Re-state the oracle in your notes — rotate to v2 → deploy → traffic on v2 → revoke v1 with audit event ROT-2026-07-30 — and keep the invariant visible: dual-key window: readers accept v1+v2; revoke v1 only after emit proves zero v1 use for 30m. Track secret_version_share and auth_fail_rate as the scoreboard. Surface under change control: secrets manager path openai/prod. If you only have forty minutes, finish the fixture for revoke v1 while indexer still cached v1 → 401 storm before polishing UI. Promotion language stays ternary: promote, hold, or roll back based on evidence, not hope.

Set a timer for twelve minutes and attempt only the observability concerns for Secrets rotation lab. Speak aloud what security engineer rotating openai/prod after contractor offboarding would see on success versus on revoke v1 while indexer still cached v1 → 401 storm. Write three bullet artifacts you must keep: a fixture name tied to SEC-REVOKE-EARLY-21, a metric query for secret_version_share and auth_fail_rate, and the rollback or refuse command for secrets manager path openai/prod. Then extend the notes with one sentence on how dual-key window: readers accept v1+v2; revoke v1 only after emit proves zero v1 use for 30m would be violated if you skipped this page. Close by restating the oracle in your own words without looking: rotate to v2 → deploy → traffic on v2 → revoke v1 with audit event ROT-2026-07-30.