Is LLM inference a cost of revenue or an R&D cost?

Recurring inference triggered by customer usage is a variable cost of serving and belongs in cost of revenue. One-off fine-tuning, prototyping, and evaluation runs generally stay in development spend.

How do you measure an AI feature's gross margin before launch?

Tag staging traffic by account, workflow, and outcome; take cost per successful task at p50, p90, and p99; multiply by expected monthly tasks per account, add paid failures and retries, and compare against plan price.

Why does average cost per request understate margin risk?

AI cost scales with usage while a flat subscription price does not, so margin failures appear in the heaviest accounts, which an average deliberately smooths away.

Quick answer: Inference is a cost of serving, not a development cost. It runs on every customer action, scales with usage, and lands in the same invoice as production infrastructure. Treating it as R&D until...

Measure an AI feature's gross margin before you ship it

Updated September 2, 2026 · first published September 2, 2026

Inference is a cost of serving, not a development cost. It runs on every customer action, scales with usage, and lands in the same invoice as production infrastructure. Treating it as R&D until launch means the first honest margin number arrives with the first full billing cycle, which is the worst possible time to learn it.

Instrument before you launch, not after

In staging, tag every model call with the account, the workflow, and the outcome, and replay realistic traffic rather than a synthetic loop. What you need out of it is a distribution: cost per successful task at p50, p90, and p99, plus the share of spend that went to failed and retried work you still pay for. A single average hides the accounts that break the margin.

Roll it up to cost per account

Multiply cost per successful task by expected tasks per account per month and add the failure share. That number is comparable to the price of the plan. Do it once at median usage, once at the 90th percentile, and once for the heaviest account you would still accept. Flat-price plans fail in the tail, because model cost keeps scaling after revenue stops.

The four cost drivers to check

Context resent per turn. A long conversation or a large retrieved document is re-billed on every turn unless caching covers it.
Retries and failures. Billed at full rate; they belong in the numerator, not in an error budget.
Agentic fan-out. One user action becomes many model and tool calls, and the multiplier is what varies between accounts.
Model routing. A default that sends easy cases to an expensive model is the largest single controllable line.

Wire the numbers to a decision

The output of the exercise should be a choice made before launch: a usage ceiling in the plan, a consumption component in the price, a cheaper default model with escalation only where quality requires it, or an accepted margin with a review date. Ship the instrumentation with the feature so the same numbers keep arriving in production and the assumption can be checked rather than defended.

Report it as its own line

Break AI cost of revenue out inside gross margin, with cost per account and margin at each usage percentile. Read that way, a rising bill is legible: spend growing in step with revenue at a flat cost per account is scale, and spend growing faster than revenue is erosion with a named cause.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research