LLM cost monitoring in production
Updated 1 August 2026
Monitoring LLM costs in production is different from monitoring traditional cloud infrastructure. The unit of cost is a token, not a server hour. A single request can cost anywhere from a fraction of a cent to several dollars. Spikes can come from prompt growth, model drift, retry loops, or one noisy customer. If your monitoring only shows the monthly invoice, you are not monitoring; you are autopsying.
The production monitoring stack
A production-ready LLM cost monitoring stack has four layers:
- Instrumentation. Capture the right fields on every request.
- Aggregation. Roll request-level data into useful dimensions.
- Alerting. Detect deviations from baseline before they compound.
- Runbooks. Tell the on-call engineer what to do when an alert fires.
Instrumentation: what to capture on every request
The minimum viable telemetry schema for cost monitoring includes:
| Field | Why it matters |
|---|---|
| Provider and exact model | Pricing differs by model version, not just family. |
| Feature or endpoint | The fastest path to actionable attribution. |
| Team or customer | Needed for chargeback and abuse detection. |
| Input tokens | Largest controllable cost driver. |
| Output tokens | Usually priced higher than input. |
| Cache-read tokens | Discounted; must be tracked separately. |
| Reasoning tokens | Billed as output; invisible to users. |
| Retries | Failed work often looks like legitimate usage. |
| Tool calls | Agents can fan out into many billed calls. |
| Estimated cost | Lets you alert before invoice close. |
OpenTelemetry's GenAI semantic conventions standardize most of these fields. Use them as the baseline rather than inventing your own schema.
Dashboards that matter
Most teams overbuild their dashboard layer. Start with five views:
- Spend by feature. Top ten endpoints or workflows by cost and trend.
- Spend by model and provider. Where expensive models are actually used.
- Spend by team or customer. The foundation of chargeback and margin analysis.
- Token trends. Input growth, output verbosity, and cache-read share over time.
- Operational ratios. Cache hit rate, retry rate, and tool-call rate.
If a dashboard does not help someone take action, remove it. Visibility without action is noise.
Alerts worth setting up
| Alert | Trigger | Why it matters |
|---|---|---|
| Daily spend by feature | Spend > 150% of rolling 7-day baseline | Catches feature-level spikes same day. |
| Unexpected model shift | Cheap endpoint suddenly routes to frontier model | Deployment or routing bug. |
| Cache-hit regression | Cache-read share drops >10 percentage points | Paying full price for reprocessing. |
| Retry storm | Median retries per task doubles | Downstream failure or brittle prompt. |
| Agent loop | Tool-call count exceeds threshold | Runaway agent cost. |
| Reconciliation variance | Internal estimate drifts >10% from provider | Missing or misclassified usage. |
Alert thresholds should be tied to baselines, not static numbers. A startup's spike is an enterprise's normal Tuesday.
Runbooks: what to do when an alert fires
Every alert needs a runbook. For LLM cost spikes, the runbook usually follows this order:
- Identify the dimension. Is it one feature, one model, one team, or one customer?
- Check for recent deploys. Did a model default change, prompt template change, or routing rule change?
- Look at token composition. Is the spike from input growth, output growth, retries, or tool calls?
- Implement a guardrail. Cap output length, switch a default model, or throttle the noisy customer.
- Validate the fix. Compare estimated cost before and after the change.
Monitoring is not the solution
Monitoring tells you where the waste is. It does not remove it. The actual savings come from the optimization work that follows: routing to cheaper models, compacting prompts, enabling caching, batching non-urgent work, and adding agent guardrails. Treat monitoring as the input to a weekly cost review, not a project that ends once the dashboard ships.
Related
- LLM cost monitoring — the foundational telemetry guide.
- Six levers for AI cost optimization — what to do after monitoring finds the waste.
- LLM cost anomaly detection — statistical methods for catching spikes.
- Agent spend guardrails — controls for runaway agent costs.
Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →
FAQ
What should I monitor for LLM costs in production?
Monitor spend by feature, model, provider, team, and customer. Track token classes separately, including input, output, cache read, reasoning, and tool-call tokens. Watch for prompt growth, model drift, retry storms, and cache-hit regressions.
How do I alert on LLM cost spikes?
Set daily spend thresholds tied to a feature's rolling baseline, not a global number. Alert on unexpected model shifts, cache-hit regressions, retry storms, and reconciliation variance between internal estimates and provider invoices.
What is the minimum telemetry schema for LLM cost monitoring?
Capture provider, model, feature or endpoint, user or team, input tokens, output tokens, cache-read tokens, reasoning tokens, retries, tool calls, and estimated cost per request. OpenTelemetry GenAI semantic conventions are a strong baseline.