LLM cost monitoring
Updated 11 June 2026 · first published 5 May 2026
LLM cost monitoring is the practice of tracking spend at request level, not just invoice level. If you cannot explain which feature, user, model, or retry pattern created the cost spike, you are not monitoring spend. You are only seeing the bill after the damage is done. Cost attribution starts here.
What to track on every request
- Provider and exact model. Model pricing changes by tier and version, so “gpt-4” is not granular enough-you need “gpt-4o” vs. “gpt-4-turbo”.
- Feature or endpoint. The fastest route to actionable attribution and savings discovery.
- User, account, or workspace. Needed for chargeback, quota control, and abuse detection.
- Input and output tokens. The primary cost driver and easiest to control with prompt or output-length adjustments.
- Cached-read tokens. OpenAI offers ~50% cache-read discount; Anthropic offers ~90%. Conflating cached and normal input tokens corrupts your baseline.
- Reasoning tokens (where applicable). Extended thinking and similar features add cost that standard token counts miss.
- Retries and tool calls. Hidden cost often sits in failed or repeated work; tool-call tokens are now a material line item.
- Estimated cost. So you can alert before invoice close, not after damage accumulates.
Token classes are billed differently - track them separately
| Token class | Billing behavior | Monitoring pitfall if conflated |
|---|---|---|
| Input (uncached) | Full list price per model | Prompt growth hides inside aggregate token counts |
| Output | Priced higher than input on most models | Verbose-output drift looks like traffic growth |
| Cache read | Discounted (~50% OpenAI, ~90% Anthropic) | Counting cache reads as normal input corrupts the baseline and overstates "savings" |
| Reasoning / thinking | Billed as output even though invisible to users | Standard token counts miss it entirely |
| Retry and tool-call | Every upstream attempt is billed, user sees one answer | Failed and repeated work reads as legitimate usage |
| Batch | Discounted asynchronous lane | Mixing batch with realtime hides which workloads could move lanes |
Use a consistent telemetry schema
OpenTelemetry's GenAI semantic conventions are a strong baseline because they standardize fields like model, input tokens, output tokens, cache-read tokens, conversation IDs, and provider names. That makes it easier to swap vendors or observability backends without rebuilding your data model every quarter. Define your schema first; the tooling (Langfuse, Helicone) will follow.
The dashboards that matter
Most teams overbuild their dashboard layer and underbuild their tagging layer. For cost attribution to work, start with these five views:
- Spend by feature. Top ten endpoints or workflows by total cost and cost trend.
- Spend by model and provider. Where the expensive models are actually being used and whether you are over-relying on one provider.
- Spend by customer or team. The foundation of chargeback and margin analysis.
- Input and output token trends. Detect prompt growth, verbose output drift, and whether cache-read benefits are real.
- Cache hit, retry, and tool-call rates. Three of the easiest places to find waste and validate optimization impact.
Alerts worth setting up
- Daily spend threshold by feature or team. Tied to your cost baseline, not a global number.
- Unexpected model shift. Example: a cheap endpoint suddenly defaults to a frontier model; catch it same day.
- Cache-hit rate regression. If your cache-read share drops, you are likely paying full price for reprocessing.
- Retry storm or agent loop behavior. Costs compound; alert on loops within a single request or session.
- Reconciliation variance. Internal cost estimates drift too far from provider truth-investigate why.
Monitoring rule
A practical stack
For most teams, provider-native billing plus a gateway layer, request metadata in application logs, and an observability tool like Langfuse or Helicone covers the essentials. For larger operations, add a warehouse layer and monthly reconciliation against provider or cloud billing exports so you can tie cost back to customer cohorts, feature releases, and margin.
Monitoring is the prerequisite, not the solution
Monitoring alone does not lower your bill. It gives you the ranked list of what to fix first. Once that list is visible, the usual wins are smaller default models, output caps, prompt cleanup, prompt caching (especially for cache-read discounts), batching, and quota controls based on unit economics.
Related
- LLM cost anomaly detection - detecting spend surges.
- AI observability - cost signals and operational metrics.
- What is LLM FinOps? - the complete operating discipline.
Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →