Should you always raise a rate limit that is being hit?

No. Check cost per completed task first. Real demand growth holds that ratio steady as volume rises; a runaway loop or retry storm breaks it, and raising the limit removes the only thing containing the spend.

Why do retries make rate limiting expensive?

Retrying on 429 without exponential backoff and jitter turns one breach into a synchronised storm. Successful retries are billed, and the amplification usually lives in a client library rather than your application code.

How should rate limits be set?

Below the provider's ceiling, separately per environment and feature, so your own limit breaks first and you control the response. Non-production should be capped hard; production caps need a deliberate owner.

Quick answer: When a service starts returning 429s, the reflex is to treat it as an availability incident: raise the limit, upgrade the tier, add retries, restore throughput. That is sometimes correct. It is also...

Rate limits, 429s and tier upgrades

Updated September 1, 2026 · first published September 1, 2026

When a service starts returning 429s, the reflex is to treat it as an availability incident: raise the limit, upgrade the tier, add retries, restore throughput. That is sometimes correct. It is also the mechanism by which a runaway loop turns into a five-figure invoice, because the only thing that was stopping it has just been removed.

What a 429 is actually telling you

It says the system is trying to spend faster than its allowance. That happens for two very different reasons, and they need opposite responses. Real demand growth — more users, a launch, a seasonal peak — means the limit is genuinely too low and raising it buys real revenue. Runaway behaviour — a retry storm, an agent looping, a backfill nobody throttled, a test suite pointed at production keys — means the limit is doing exactly its job.

The two are indistinguishable from the error rate alone. They are trivially distinguishable if you look at cost per unit of work: real growth holds the ratio roughly constant while volume rises; a runaway breaks it, spending far more per completed task than yesterday. That single ratio should gate every tier upgrade.

Retries make it worse, quietly

Naive retry on 429 without exponential backoff and jitter converts one limit breach into a synchronised storm. The requests that succeed are billed; many of the ones that fail have still consumed input processing. And because retries are usually implemented in a client library rather than application code, the amplification does not appear anywhere in the feature's design docs.

Use limits deliberately

Set your own limits below the provider's, per environment and per feature, so the first thing to break is yours and you control what happens next. Keep non-production strictly capped — a failed test build is cheap, a capped customer path is not. Alert on the cost-per-task ratio, not just on error counts. And when you do upgrade a tier, treat it as a spending decision with a named owner, because that is what it is.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research