Inject a controlled runbooks failure

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

This page deliberately breaks Runbooks in a disposable or explicitly approved environment. The scenario is inject one poison PDF that retries forever plus provider 429s; the operator must quarantine only the poison message, lower concurrency during throttling, and avoid purging the queue. Write the hypothesis as an ordered signal chain: fault injection, component-level signal, user-visible indicator, protective action, and recovery. Include expected timestamps or maximum delays. “The system becomes unhealthy” is not precise enough to debug.

Capture a baseline of embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs and record the effective configuration:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Define hard aborts before starting: unexpected data mutation, impact outside the test scope, unbounded cost, missing telemetry, or inability to remove the fault. Name the person who can stop the drill. Confirm the cleanup command and take a state snapshot that proves the environment can be restored.

Run exactly one fault workflow:

./ops/game-day embedding-backlog --poison-doc fixture-900mb.pdf --provider-429 0.30
./ops/queue quarantine --message "$MESSAGE_ID" --simulation

Mark the start time in UTC. Do not tune the system during the first observation window. Compare actual signal order with the hypothesis and note the first divergence. The target remains detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. A protection mechanism may intentionally reject or degrade some work; count that as success only if the user contract explicitly permits it and unaffected traffic remains inside its objective.

Expected telemetry must show demand as well as outcomes. A flat error counter during a traffic outage is not resilience. Verify the injector itself worked through an independent observation such as dependency latency, worker count, policy state, or fault-proxy statistics. If no signal moves, stop and diagnose the drill rather than escalating fault intensity blindly.

Remove the injection at the planned time, preserve logs, and observe several consecutive samples. Check backlog, delayed retries, stale endpoints, cached state, and circuit or escalation state; recovery of the front-door status alone is insufficient. Compare final state to the baseline and explain any residual difference.

Use the historical incident as a reason to be exact: a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal. The production tradeoff is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. State whether the current behavior contains that risk or merely moves it. The evidence bundle must contain the hypothesis, baseline, fault command, start and stop times, telemetry queries, cleanup proof, and one finding that changes a future design or operating decision.