How much of an LLM bill is typically non-production?

It varies, but teams that measure it are routinely surprised to find a double-digit percentage never served a customer. CI evaluation suites are usually the largest single contributor, followed by long-lived staging instances.

What is the first thing to change?

A separate API key per environment, with the environment tagged on every request. It takes an afternoon and turns one unexplained line item into labelled ones, which is a precondition for every other fix.

Is it safe to put a hard spend cap on non-production?

Yes, and it is the one place a hard cap is genuinely safe. A capped test environment fails a build, which someone notices and fixes; a capped production environment fails a customer.

Quick answer: Every FinOps review of LLM spend starts with production traffic, because that is what has a dashboard. Then someone runs the numbers by environment and finds that a meaningful slice of the bill...

Non-production LLM spend

Updated September 1, 2026 · first published September 1, 2026

Every FinOps review of LLM spend starts with production traffic, because that is what has a dashboard. Then someone runs the numbers by environment and finds that a meaningful slice of the bill — frequently double digits as a percentage — never touched a customer.

It comes from four places, and they compound.

Where non-production spend comes from

CI. Every pull request that runs an evaluation suite calls the model. A suite of two hundred cases across a busy repository is thousands of calls a day, on the most capable model, because whoever wrote the suite quite reasonably wanted it to reflect production.

Local development. Engineers iterating on a prompt call the API dozens of times an hour, usually on a shared key, usually with no tagging that says whose loop it was.

Staging and demo instances. Long-lived, low-traffic, and configured identically to production — which means the expensive model and no caching, serving a handful of requests that nobody watches.

Retries and runaway loops in test. An agent loop that fails safely in production still burns tokens when it misbehaves in a branch, and nothing in a test environment is set up to notice.

Fix attribution before you fix cost

The single change that pays for itself is a separate API key per environment. It costs an afternoon and turns an unexplained line item into four labelled ones. Tag every request with the environment, and for CI, the repository and workflow. Without that, every proposal below is guesswork.

Once you can see it, three things usually follow immediately. Non-prod is where a smaller model is almost always fine — evaluation suites need consistency, not frontier capability, and most of them can pin a cheaper model with no loss of signal. Non-prod is where caching earns the most, because CI replays near-identical prompts all day. And non-prod is where hard spend caps are actually safe to set: a capped test environment fails a build, while a capped production environment fails a customer.

The rule

Give every non-production environment its own key, its own budget, and its own cap. Then treat the gap between environments as a number you manage deliberately, rather than one you discover in a quarterly review.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research