How to cap inference costs and prevent runaway spending

A cap on inference spending is a hard limit that stops new requests when cost reaches a threshold, preventing runaway bills from agent loops, misconfigured retries, or unexpected traffic spikes. Caps exist at three levels: the API provider (budget caps on your account), the gateway or proxy layer (per-key or per-team limits), and the application (per-run token budgets, agent step limits, and idle-instance autoscaling). Most cost explosions can be prevented by implementing hard caps at all three, paired with alerting that fires before the cap is reached.

This page covers the mechanics of capping costs, when each cap works, and the tradeoffs between hard stops and graceful degradation.

Provider-level budget caps

OpenAI, Anthropic, and most other providers offer a usage limit or spend cap at the organization level. OpenAI allows you to set a hard maximum monthly spend in the billing settings; requests fail with a quota-exceeded error once that limit is reached. Anthropic supports per-API-key monthly budgets with similar behavior. Azure OpenAI has deployment-level rate limits and quota caps. Bedrock provides account-level usage limits.

The advantage of a provider cap is simplicity - it is the easiest lever to pull to prevent a runaway month. The disadvantage is coarseness: when the cap is hit, all traffic stops, whether it is mission-critical or experimental. If you hit a monthly cap on July 15, critical features are dark until August 1.

How to set it: visit your provider's billing console, look for "usage limits," "spend caps," or "budget alerts," and set a number. Set it as a safety net, not as your primary control - alerts should fire long before the cap triggers.

Gateway and per-key budget caps

A gateway or API proxy layer sits between your application and the provider, routing requests and enforcing per-key or per-team budgets. Tools like LiteLLM, Ollama with custom middleware, and commercial proxies (such as those offered by enterprise observability platforms) can tag requests by team, feature, or environment, track spend per tag, and reject or degrade requests when a tag-specific budget is exceeded.

A gateway cap is more granular than a provider cap. You can set different budgets for different teams: $500/month for team-support, $2000/month for team-search, $100/month for experiments. When team-support hits its limit, requests from team-support are rejected or downgraded to a cheaper model, but team-search continues normally. This prevents a single team's runaway feature from blocking everyone else.

The tradeoff is complexity: you now have to deploy and maintain a gateway, tag requests consistently, and monitor per-team spend. But the control is worth it in any multi-team environment.

How to implement: deploy a gateway that reads API keys or request headers, maps them to teams/features, and maintains a spend ledger per tag. Reject requests (HTTP 429) when a tag's daily or monthly budget is exceeded, or degrade to a cheaper model instead of rejecting. Pair this with a dashboard that shows spend per tag, so teams can see their burn rate in real time.

Request rejection vs. graceful degradation

When a cost cap is hit, you have two choices: fail the request (rejection) or transparently degrade (use a cheaper model or return a cached result).

Rejection is the safest cap. A request hits the budget limit and returns an error immediately. The application knows it failed and can decide what to do: show an error to the user, queue for retry, or fall back to a non-AI feature. Rejection guarantees you will not exceed the budget, but it hurts availability.

Degradation is the most user-friendly cap. A request hits the budget limit, but instead of failing, the gateway silently routes it to a cheaper model or returns a cached result. The user sees no error and the feature works, just potentially with lower quality or freshness. Degradation protects availability but requires careful tuning: if the cheaper model is too much worse, users will notice the drop in quality and blame the feature, not the cost control.

Best practice is to use both: hard caps at the organization level (rejection, to prevent catastrophic overspend) and graceful degradation at the feature level (cheaper model, cached results, or partial feature disabling, to protect user experience). Pair both with alerts at 50%, 75%, and 90% of budget so engineers have time to fix the underlying issue before either cap kicks in.

Runaway agent kill switches and token budgets

Agent workloads are the most dangerous to costs because they can loop indefinitely. A planning bug, a tool that returns noise, or a missing termination condition can cause an agent to make dozens of model calls for a task that should take three. Prevent this with per-run safeguards.

Per-run token budget: every agent invocation carries a maximum total input + output token limit. If the run exceeds it, the agent stops and returns an error. This is the primary circuit-breaker. Set it tight enough to catch loops (e.g., 100K tokens for a typical task) but not so tight that normal tasks fail.

Per-run cost budget: absolute dollar cap per agent run. Especially critical when an agent can escalate from a cheap model to an expensive one mid-execution on failure. If the run would cost more than $1, stop and return an error instead of continuing.

Step and loop limits: an agent should not be allowed to run more than N planning steps (e.g., 20) or more than M tool calls per run (e.g., 50). Either limit indicates a logic error or a pathological input the agent cannot solve.

Tool-call allowlists: only allow the agent to invoke tools that have been explicitly registered. Add per-tool invocation caps (e.g., "call search tool at most 5 times per run"). Prevent unexpected or exploratory tool discovery.

Kill-switch: an explicit per-agent feature flag that on-call engineers can flip to disable the agent entirely without a deploy. When an agent becomes unstable (due to a prompt change, a tool latency regression, or a bug), the kill-switch lets the team stop the damage immediately. The switch should be observable in dashboards so the team knows it is active and knows to fix the underlying problem.

Idle GPU autoscaling and right-sizing

Self-hosted GPU infrastructure incurs cost whenever instances are running, even if no requests are being processed. A small batch of inference work during business hours does not need a 4xA100 cluster sitting idle overnight. The fix is autoscaling.

Autoscale to zero: when the queue is empty and no request has been received in N minutes, scale instances down to zero. This eliminates idle-time cost. The tradeoff is cold-start latency for the first request after a scale-down, but 100ms of cold start is almost always acceptable.

Right-size capacity: measure your p99 request latency and queue depth. If p99 latency is still acceptable with fewer instances, you are paying for idle capacity. Many teams overprovision GPU capacity "just in case," then never use it. Start conservative, measure, and only add capacity when latency degrades.

Use spot or preemptible instances: spot instances (AWS, GCP, Azure) are 60–80% cheaper than on-demand, but can be interrupted. Pair spot with on-demand fallback and persistent queues so interrupted jobs can retry. This trades reliability risk for cost; acceptable for batch work and evals, riskier for synchronous inference.

Measure idle time: track the fraction of time instances are running with zero or near-zero requests. If an instance is idle more than 20 percent of the time, right-size down. If idle time is below 5 percent, consider adding capacity to reduce queue depth.

Alerts vs. hard caps: the defense in depth strategy

Hard caps prevent surprises, but they also stop traffic when hit. Alerts are the first line of defense: they fire when spend is 50 percent of budget, giving you time to investigate and fix the underlying issue before the hard cap is triggered. Hard caps are the circuit-breaker when alerts fail or when a cost event is happening faster than humans can respond.

A complete strategy:

The goal is never to hit the hard cap. It is there as a circuit-breaker for when monitoring fails or when a cost event is happening too fast for humans to respond. In a well-run FinOps practice, hard caps almost never trigger.

Related

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research