Define the production target for runbooks

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

Start with the request or operator journey, not a tool choice. For Runbooks, draw worker queue document-embedding entering an embedding ingestion pipeline, then mark every place where work can queue, fail, or return a misleading success. Label the actor who experiences each outcome. The target is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. Rewrite that sentence as an observable promise and list two non-goals so a later engineer cannot quietly expand the scope.

The acceptance contract is: detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. Break it into a table in your notes with columns for indicator, threshold, window, data source, and owner. Ratios require an eligible-event definition; latency targets require a start and stop boundary; recovery objectives require durable checkpoints. State whether retries, synthetic probes, client cancellation, fallbacks, and maintenance are included. If eligibility is ambiguous, two correct implementations can report incompatible reliability.

Use this incident as the concrete threat model: a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal. Separate the trigger, contributing conditions, user impact, and delayed detection. Then rank three failure modes by likelihood and impact. The planned drill is inject one poison PDF that retries forever plus provider 429s; the operator must quarantine only the poison message, lower concurrency during throttling, and avoid purging the queue. Explain why that fault is representative and why its scope is safe enough for staging.

The essential signals are embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs. For each, write its unit, expected baseline, threshold, and maximum observation delay. Counters should be monotonic; histograms need buckets around the objective; gauges need an interpretation when traffic is zero. Do not put prompts, document text, raw user IDs, or other unbounded values in labels.

Treat the following as a design proposal, not accepted production truth:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Review every number against demand and consequence. highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. The lab owner must document which side of that tradeoff is preferred, when the preference expires, and what evidence would justify changing it. Add an abort condition for cost, data integrity, unrelated tenants, and failed rollback.

Use this command only to inspect or establish the initial baseline:

./ops/runbook lint runbooks/embedding-backlog.md
./ops/runbook links runbooks/embedding-backlog.md --check-local

Record UTC time, environment, build or policy revision, expected output, and actual output. A screenshot without query text and timestamp is not durable evidence. The page is complete when another engineer can explain what is protected, what is deliberately not protected, and how success can be independently calculated.