Canary and operate runbooks

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

Before shipping Runbooks, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./ops/runbook publish embedding-backlog --owner search-platform
./ops/runbook review-due embedding-backlog --days 90

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: inject one poison PDF that retries forever plus provider 429s; the operator must quarantine only the poison message, lower concurrency during throttling, and avoid purging the queue. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal.—as a regression scenario. The enduring production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.