Glossary

Plain-English definitions of the terms used across AI FinOps, LLM cost attribution, and optimization. Listed alphabetically.

AI FinOps: Applying visibility, attribution, optimization, and accountability practices to AI and LLM spend, so cost behaves like a managed unit-economics input rather than a surprise on the invoice.
Anomaly detection: Flagging unusual changes in LLM spend - a sudden spike in tokens, a model-mix shift, or a runaway agent loop - early enough to act before the monthly invoice arrives.
Attribution (cost attribution): Mapping each unit of LLM spend back to the feature, team, environment, and customer that caused it, so every dollar has a named owner.
Baseline: A locked, pre-optimization measurement of spend used as the reference point for measuring savings. Reconciled to the provider invoice so reported savings are defensible.
Batch API: An asynchronous request mode offered by major providers, typically at a discount versus real-time calls, for work that can tolerate delayed completion (evals, backfills, summaries).
Budget enforcement: Turning a monthly spend envelope into an actual control: soft thresholds alert the owning team; hard thresholds degrade behavior (cheaper model, less retrieval, queue or batch) rather than letting spend run unbounded.
Cache-read tokens: Input tokens served from a provider's prompt cache rather than processed fresh. They are billed at a steep discount (often around 50% on OpenAI and up to ~90% on Anthropic, subject to current provider pricing), so tracking them separately is essential to an accurate baseline.
Chargeback: Assigning spend to the teams, products, or customers that consumed it and moving the cost onto their budget or P&L.
Context window: The maximum number of tokens a model can consider at once (prompt plus generated output). Larger contexts cost more per request and are a common, often invisible, driver of bill growth.
Cost per request: The fully-loaded cost of a single LLM call - input, output, cache, and reasoning tokens - used as a product KPI to track the unit economics of a feature over time.
Fallback chain: An ordered list of models a request can fall back to when the primary is unavailable, rate-limited, or too expensive. Improves reliability but can quietly raise cost if a pricier fallback fires often.
Gross margin (AI feature): Revenue from an AI-powered feature minus the LLM and infrastructure cost to deliver it. The number AI FinOps ultimately exists to protect.
Input tokens: The tokens you send to a model - the prompt, system instructions, retrieved context, and conversation history. Usually cheaper per token than output, but easy to inflate with oversized prompts.
Invoice reconciliation: The monthly close that ties internal usage estimates (from gateway or log data) line-by-line to the provider invoice, documenting and fixing any variance.
Latency budget: The maximum acceptable response time for a request. It bounds which optimizations are viable - some routing and batching choices trade latency for cost.
Model routing: Sending each request to the cheapest model or provider that still satisfies quality and latency constraints, instead of defaulting every call to the most capable model.
Output tokens: The tokens a model generates in its response. Typically the most expensive token type per unit, so verbose outputs and high max-token limits are frequent cost drivers.
Prompt caching: A provider feature that stores a repeated prompt prefix so subsequent calls reuse it at a discount. Effective for stable system prompts and shared context.
Prompt compression: Reducing the token count of prompts - trimming boilerplate, summarizing history, pruning retrieved context - to lower cost without materially changing output quality.
Provider arbitrage: Running the same or an equivalent model across providers and routing to whichever offers the best price for the required quality and latency.
Quality SLO: A measurable threshold (accuracy, eval score, human rating) that an optimization must hold above. Optimizations that breach the SLO are rolled back rather than shipped for the sake of savings.
RAG (retrieval-augmented generation): Injecting retrieved documents into a prompt so the model answers from current or proprietary data. Retrieval depth directly drives input-token cost, making it a key attribution and optimization lever.
Reasoning tokens: Tokens a reasoning model generates internally while "thinking" before producing a visible answer. They are billed but not shown, and can dominate the cost of a request - a frequent source of unexplained bill growth.
Semantic caching: Reusing a stored response for a semantically similar request once it passes a similarity and quality threshold, avoiding a fresh model call entirely.
Showback: Reporting each team's spend ownership monthly without moving the cost onto their budget - visibility without billing.
Token: The unit providers bill on - a chunk of text roughly a few characters long. All LLM cost ultimately reduces to how many tokens of each type a workload consumes.
Unit economics: Cost and revenue expressed per unit - per request, per feature, per customer - so AI spend can be reasoned about the way every other variable cost is.
Unallocated spend: LLM cost that has not been traced to an owner. Treated as a bucket with a named owner and driven toward zero, rather than ignored as "miscellaneous."

Back to FinOps LLM