Secure rate limiting operations

You own a multi-tenant RAG answer API at POST /v1/answers. The goal is to protect shared model and vector-search capacity without punishing normal interactive users. The measurable target is a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After. The known production tension is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions.

List every capability needed to configure, inspect, inject failure into, mitigate, and roll back Rate limiting for a multi-tenant RAG answer API. Separate read, change, and destructive permissions. Grant them to short-lived roles rather than permanent personal credentials. The ordinary workflow must not require administrator access, and emergency access must expire automatically and emit an audit event.

Review this operational configuration for dangerous defaults:

limits:
  user: { rate_per_minute: 60, burst: 10 }
  tenant: { rate_per_minute: 1200, burst: 100 }
  global: { rate_per_minute: 4000, burst: 250 }
backend:
  redis_timeout_ms: 40
  on_timeout: local_bucket
response:
  status: 429
  retry_after: true

Check fail-open versus fail-closed behavior, external endpoints, data retention, tenant isolation, command scope, and rollback authorization. Validate unknown fields and unsafe ranges. If a setting can affect all tenants, require staged application and an independent approver. Keep secrets in the platform's secret store; never print them in config dumps, command history, metrics, or drill artifacts.

Run the security and operations gate:

kubectl apply -f deploy/rate-limit-config.yaml
kubectl rollout status deploy/answer-api --timeout=120s

Prove a read-only responder cannot perform the mutation, an authorized role can perform only the scoped change, and the audit record contains actor, UTC time, target, revision, and result. Then revoke or expire the role and repeat the denial check. For scripts, pin dependencies, quote user-controlled values, enable strict error handling, and provide a dry-run mode for high-impact operations.

The monitoring surface rate_limit_decisions_total{scope,outcome}, rate_limit_redis_seconds, http_requests_total{status="429"}, and allowed request p95 latency must avoid raw prompts, document contents, credentials, email addresses, and unconstrained customer identifiers. Use reason-code enums and controlled dimensions. Restrict detailed logs by role and retention, and test redaction with deliberately planted synthetic secrets.

The system must still meet a 60 requests/minute user bucket with a burst of 10, a 1,200 requests/minute tenant ceiling, and a 4,000 requests/minute global guard; rejected requests return HTTP 429 in under 25 ms with Retry-After. under security constraints; a control that makes mitigation impossible during an outage is incomplete. Exercise the scenario run a retry storm from one tenant at 180 requests/second while Redis latency is injected at 250 ms; verify the global guard remains available and the limiter fails closed only for expensive generation. with the least-privilege role and confirm rollback remains possible if a dependency or identity provider is degraded. Estimate provider, compute, storage, and telemetry cost, and set a hard drill budget where applicable.

The prior incident was tenant acme-importer shipped a zero-jitter retry loop, consumed 74% of the global model budget, and pushed interactive p95 from 1.8 s to 6.4 s. Ask whether excess access, missing approval, poor auditability, or unsafe tooling could have worsened it. The core tradeoff is strict per-user fairness prevents a noisy neighbor but can block legitimate batch imports; local fallback buckets improve availability but permit bounded overshoot during Redis partitions. Document who accepts the residual risk and when it will be reviewed.

Approval requires threat model, permission matrix, redaction test, audit sample, rollback owner, cost bound, and evidence that emergency access works and expires. Do not approve on policy prose alone.