Run the runbooks baseline

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

Build the smallest normal path through worker queue document-embedding for an embedding ingestion pipeline. Use a fixed, non-sensitive fixture and assign a correlation identifier at ingress. The purpose is to establish known-good semantics before fault injection. For Runbooks, the path must contribute directly to this goal: create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. Avoid adding retries, fallback branches, or tuning until the baseline is observable.

Use the reviewed configuration:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Run the baseline command:

./ops/runbook execute embedding-backlog --environment staging --dry-run
./ops/queue oldest document-embedding --limit 5

Record response status or operator state, important headers or fields, elapsed time, and side effects. Follow the same correlation identifier through structured logs. If the system is asynchronous, capture enqueue time, start time, completion time, retry count, and durable checkpoint. Run at least five samples so one warm cache or connection setup does not masquerade as normal behavior.

The acceptance target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. Query embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs immediately before and after the baseline. Calculate deltas rather than trusting dashboard shape. Every successful user event should map to a defined outcome counter; every duration should have a clearly named clock boundary. Confirm version and environment dimensions let an operator distinguish old and new code without adding customer-specific cardinality.

Write an expected event sequence before looking at logs. Compare it to the observed sequence and explain every extra event, especially hidden retries or duplicate processing. A successful response with unexpected retries is operational debt because it consumes capacity and predicts a sharper failure under load. Verify that logs contain reason codes and correlation IDs but not request bodies, credentials, or private document content.

Vary one legitimate input dimension—request size, tenant class, queue age, or dependency latency—and keep the rest fixed. Identify where the normal path approaches the target. Save p50, p95, and maximum for a bounded sample, plus demand and saturation. The baseline is invalid if traffic count is missing.

Use the historical failure as a warning: a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal. Explain which normal-path measurement would have made that incident easier to recognize. Also record the production tension: highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. The baseline page passes when another engineer can reproduce the journey and predict which telemetry changes before running it.