Test runbooks signals and thresholds

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

Convert the Runbooks objective into tests, not screenshots. The target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. Write one assertion for the exact boundary, one immediately inside it, and one immediately outside it. Include denominator behavior when traffic is zero, missing, duplicated, delayed, or partially sampled.

The telemetry contract is embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs. For every series, verify metric type, unit, label allowlist, ownership, and retention. Counters must only rise; histograms need buckets around the target; gauges require staleness handling. Confirm dashboards display demand, successful or protected outcomes, user latency or age, saturation, and version context together. A green ratio without its event count is not sufficient evidence.

Run the test workflow:

pytest -q tests/test_runbook_commands.py
./ops/runbook drill embedding-backlog --assert-diagnosis-under 480

Store machine-readable output as a build artifact. Seed deterministic time or fixtures where possible, but keep at least one integration test against the actual collector and rule engine. Tests must fail when the threshold expression, label name, or alert receiver is intentionally broken. A test that remains green after its signal is disconnected is worse than no test because it creates false confidence.

Use the configured contract:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Replay the controlled scenario—inject one poison PDF that retries forever plus provider 429s; the operator must quarantine only the poison message, lower concurrency during throttling, and avoid purging the queue.—and measure time from fault start to metric visibility, alert pending, alert firing, delivery, acknowledgement where relevant, mitigation, and stable recovery. Compare those times with the operational objective. Verify short benign blips do not page if the policy says they should not, while sustained or fast-burning impact does.

Check telemetry loss separately from service loss. Stop or misconfigure one scrape target and prove the system reports missing data rather than silently treating it as healthy. Verify logs and drill artifacts redact secrets and private payloads. Bound label cardinality using a representative workload and set an alert or budget for unexpected series growth.

Use the prior incident—a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal.—to ask which test would have detected the problem earlier. Remove or demote signals that do not lead to a decision. The production tradeoff is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. Quantify the cost of collection and paging against the risk reduction it buys.

This page passes only when contract tests, failure tests, alert-path tests, and recovery assertions all run from a clean environment and produce retained evidence linked to the configuration revision.