Secure runbooks operations

You own an embedding ingestion pipeline at worker queue document-embedding. The goal is to create an executable, decision-oriented runbook for queue backlog that a responder unfamiliar with the pipeline can use safely. The measurable target is detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. The known production tension is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Runbooks for an embedding ingestion pipeline. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

runbook:
  trigger: embedding_queue_oldest_age_seconds > 600 for 5m
  owner: search-platform
  prerequisites: [queue-read, worker-scale, dlq-write]
  guardrails:
    max_concurrency_change_percent: 25
    forbid: [queue-purge, unbounded-replay]
  success: oldest_age_declines_three_samples

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

./ops/runbook permissions runbooks/embedding-backlog.md --least-privilege
./ops/runbook danger-scan runbooks/embedding-backlog.md

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface embedding_queue_oldest_age_seconds, embedding_jobs_total{outcome}, provider_responses_total{status}, worker_concurrency, dead_letter_queue_depth, and document_id in structured logs must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet detect queue age above 10 minutes, diagnose provider throttling versus poison documents in under 8 minutes, cap concurrency changes at 25%, and verify queue age declines for three consecutive samples before closing. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario inject one poison PDF that retries forever plus provider 429s; the operator must quarantine only the poison message, lower concurrency during throttling, and avoid purging the queue. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was a malformed 900 MB PDF retried 4,200 times, held a FIFO shard, and drove oldest-message age to 47 minutes while aggregate queue depth looked normal. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is highly prescriptive commands reduce cognitive load but become dangerous when flags drift; flexible diagnosis survives change but demands more expertise during stress. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.