Canary and operate rate limiting

You own a multi-tenant RAG answer API at POST /v1/answers. The goal is to protect shared model and vector-search capacity without punishing normal interactive users. The measurable target is a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After. The known production tension is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions.

Before shipping Rate limiting, link the goal, reviewed configuration, baseline, fault drill, diagnostic timeline, automated tests, security review, and rollback procedure. The production objective is a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After. Every claim in the release note must point to a command output, telemetry query, or approved decision. Missing evidence is a release blocker, not an item to infer from confidence.

The candidate configuration is:

limits:
  user: { rate_per_minute: 60, burst: 10 }
  tenant: { rate_per_minute: 1200, burst: 100 }
  global: { rate_per_minute: 4000, burst: 250 }
backend:
  redis_timeout_ms: 40
  on_timeout: local_bucket
response:
  status: 429
  retry_after: true

Diff it against the running revision and identify any field with fleet-wide effect. Name the operator, approver, observation window, and rollback trigger. Confirm dashboards and alerts query the candidate's labels before sending traffic. Freeze unrelated changes during the canary so attribution remains possible.

Run the staged rollout workflow:

./scripts/canary-limiter.sh --traffic 5 --minutes 15
./scripts/verify-limiter-slo.sh --window 30m

Start with the smallest representative slice that can reveal the known failure mode. Compare candidate and control on demand, outcomes, latency or age, saturation, cost, and the primary series rate_limit_decisions_total{scope,outcome}, rate_limit_redis_seconds, http_requests_total{status="429"}, and allowed request p95 latency. Observe longer than the slowest timeout, queue cycle, probe threshold, escalation interval, or data-rebuild checkpoint relevant to this lab. Promote only on prewritten criteria; do not move a threshold after seeing inconvenient data.

Abort and roll back if the controlled risk appears: run a retry storm from one tenant at 180 requests/second while Redis latency is injected at 250 ms; verify the global guard remains available and the limiter fails closed only for expensive generation. After rollback, prove configuration revision, traffic allocation, deferred work, and user indicators returned to baseline. Keep the evidence even when the canary succeeds so the next operator has a reference distribution.

Publish owner, escalation path, runbook, dashboard, alert meaning, safe commands, access prerequisites, and review date. Schedule the next game day and define what architectural change invalidates this procedure. Track near misses, pages, manual interventions, false positives, cost, and time to mitigation over a 28-day window. A shipped control that nobody reviews will drift as traffic and dependencies change.

Use the historical incident—tenant acme-importer shipped a zero-jitter retry loop, consumed 74% of the global model budget, and pushed interactive p95 from 1.8 s to 6.4 s.—as a regression scenario. The enduring production tension is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions. State what the rollout chooses today and what metric would force reconsideration. Close the release only when the on-call owner accepts the handoff and can execute rollback without the implementation author.