AI Agent Rate Limiting Strategy 2026
An agent with no rate limiting can burn a month of quota in ten seconds. This guide covers three layers - concurrency, rate and budget - with concrete configuration and a formula for sizing them.
💡 What You Will Learn
An agent with no rate limiting can burn a month of quota in ten seconds. This guide covers three layers - concurrency, rate and budget - with concrete configuration and a formula for sizing them.
📜 Table of Contents
How fast an unlimited agent burns a quota
The pattern is always the same: a batch job starts, the agent fires at full concurrency, 429s appear within seconds, the account gets throttled, and the invoice arrives over budget. No line of code is wrong - nobody put a valve on the outlet.
Rate limiting needs three layers: concurrency stops instant overload, rate stops sustained quota overrun, budget stops a single runaway task. Together they cap both performance and spend.
The three layers
| Layer | Controls | Unit | Behaviour on trigger |
|---|---|---|---|
| Concurrency | Tasks in flight | count | Queue and wait |
| Rate | Calls or tokens per interval | calls/min, tokens/min | Wait or reject |
| Budget | Cumulative spend | per day/month | Stop and alert |
Layer 1: Concurrency
import asyncio
sem = asyncio.Semaphore(5)
async def run_one(task):
async with sem:
return await agent.run(task)
Size the semaphore to the provider's concurrency limit, not to how many coroutines your machine can spawn. Add a per-task timeout so one stuck task does not hold a slot forever.
Layer 2: Rate limiting
Concurrency alone cannot cap "total calls per minute", because one task may call the API many times. Token bucket (allows bursts) and sliding window (stricter) are the two common algorithms. In multi-process or multi-instance deployments the counter must live in shared storage such as Redis, or each process keeps its own and the aggregate is N times the limit.
Always honour the server's backoff signal: on 429, read Retry-After when present, otherwise retry with exponential backoff plus jitter. Retries without jitter produce a thundering herd that re-saturates the quota the moment it recovers.
async def call_with_backoff(fn, max_retries=4):
for i in range(max_retries):
try:
return await fn()
except RateLimited as e:
await asyncio.sleep(e.retry_after or (2 ** i) + random.random())
raise RuntimeError("rate limited after retries")
Layer 3: Budget circuit breaker
Two jobs: account for every call (input and output tokens, converted to cost, accumulated daily) and trip when the daily or monthly ceiling is crossed. Accounting must cover streaming, tool calls and retries. Also set a per-task cost ceiling - a runaway loop can eat a daily budget in minutes.
Sizing the numbers
Do not copy someone else's constants; derive them from your own quota.
| Target | Formula |
|---|---|
| Calls per minute | Token quota per minute / average tokens per call |
| Max concurrency | Calls per minute x average latency (s) / 60, capped by provider limit |
| Per-task budget | Daily budget x expected task share (typically 1-5%) |
Tune against reality: a sustained 429 rate above 1% means the rate is too high; queue waits of many minutes mean it is too low.
Pitfalls
- Limiting only the front end while batch jobs bypass the limiter
- Per-instance counters in a multi-instance deployment
- Fixed-interval retries creating synchronised storms
- Streaming responses whose tokens are never counted
- Limiting call count but not cost
FAQ
Concurrency or rate first? Concurrency - a few lines stop the most common failure.
The SDK already retries, so do we need this? Yes. SDK retries handle one failed request, not total volume exceeding a quota.
What budget ceiling? Start at about 1.5x your daily budget and tighten after a week of data, and always add a per-task cap.
Is a long queue a problem? It looks like "nothing happened" to users. Add priorities for interactive requests, or surface queue state to the UI.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.
