Triage and recover rate limiting

You own a multi-tenant RAG answer API at POST /v1/answers. The goal is to protect shared model and vector-search capacity without punishing normal interactive users. The measurable target is a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After. The known production tension is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions.

Assume the bounded failure is active: run a retry storm from one tenant at 180 requests/second while Redis latency is injected at 250 ms; verify the global guard remains available and the limiter fails closed only for expensive generation. Begin with the user-visible objective—a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After.—and verify demand is present. Classify severity from measured impact, not from the apparent prestige of the dependency. Create a UTC timeline and record every observation separately from every hypothesis.

Query rate_limit_decisions_total{scope,outcome}, rate_limit_redis_seconds, http_requests_total{status="429"}, and allowed request p95 latency by stable dimensions: environment, version, outcome, dependency, and instance or region where cardinality permits. Compare an affected slice with a healthy control. Narrow in this order: deployment/configuration change, tenant or workload shape, dependency response, resource saturation, then individual instance. That order reduces random restarts and preserves useful evidence.

Use the diagnostic workflow:

redis-cli --latency-history
curl -s 'http://localhost:9090/api/v1/query?query=sum(rate(rate_limit_decisions_total%7Boutcome%3D%22deny%22%7D%5B5m%5D))' | jq .

Before each command, write what result supports the current hypothesis and what result falsifies it. A query that cannot change your decision is noise during an incident. Preserve correlation identifiers and exact query ranges. Do not paste secrets or customer content into the timeline.

The relevant operational configuration is:

limits:
  user: { rate_per_minute: 60, burst: 10 }
  tenant: { rate_per_minute: 1200, burst: 100 }
  global: { rate_per_minute: 4000, burst: 250 }
backend:
  redis_timeout_ms: 40
  on_timeout: local_bucket
response:
  status: 429
  retry_after: true

Prefer a reversible action scoped to the failing dimension: disable one flag, quarantine one message, reduce one concurrency pool, route one tenant, or select a verified recovery artifact. Give the mitigation an owner and expiry. Scaling, restarting, widening a timeout, suppressing an alert, or purging state may hide symptoms while increasing cost or destroying evidence; require an explicit reason before using them.

After the action, verify both harm reduction and side effects across several samples. Check deferred work and retries so the incident does not recur when a queue drains or a circuit closes. If the metric recovers but demand vanished, mitigation is unproven.

The historical case was tenant acme-importer shipped a zero-jitter retry loop, consumed 74% of the global model budget, and pushed interactive p95 from 1.8 s to 6.4 s. Build a causal tree with at least two competing explanations, and identify the observation that eliminated each losing branch. Do not claim root cause solely because rollback correlated with recovery. Preserve build version, configuration revision, dependency status, and relevant state for later analysis.

The design tension is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions. Record whether mitigation favored availability, correctness, fairness, cost, or responder safety. This page passes when a second responder can reconstruct why the selected action was proportionate and why broader actions were rejected.