Rate limiting lab
**Throttle, backoff, circuit breakers** — protect users, budgets, and upstream APIs.
Reviewed July 2026 · interactive lab
1Try it yourself
Playground
Rate limit lab
Protect your app and wallet — throttle, backoff, or open the circuit.
1000 req/s burst on /api/chat
2Read & reflect
Recap
#What you just did
You picked throttle vs exponential backoff vs circuit breaker for three failure shapes.
Teach
#Pattern
# Per-user token bucket + global cap
if not bucket.allow(user_id, tokens=estimate):
return 429, {"retry_after": bucket.retry_after(user_id)}
Pair with exponential backoff + jitter on provider 429/503 and a circuit breaker when embeddings stay down.
Use it
#When you'd use this
- Public chat endpoints and embed APIs
- Batch jobs that retry on failure
- Multi-tenant SaaS with usage tiers
Watch out
#Watch out
Hard infinite retries on bad JSON from tools will burn budget — cap attempts.
Try next
#Try this next
Define per-user and global limits for a 10k MAU RAG app.
3Spark check