Does stopping a streaming response stop model billing?

Not necessarily. Closing a tab or stopping the renderer does not guarantee that the upstream model request was cancelled. Propagate disconnects and explicit stop actions through the gateway, then reconcile generated, delivered, and billed tokens.

How should abandoned streaming output be measured?

Track tokens generated upstream minus tokens delivered to the client, then report the difference by feature, model, device, and cancellation reason. Also measure its dollar cost because a few long generations can dominate the waste.

What is a better KPI than cost per request for streaming AI?

Use cost per delivered response and cost per successful task, with abandonment rate beside them. Cost per request hides whether the user received the output and whether the work produced business value.

Quick answer: Streaming makes an AI product feel fast, but it also makes an unfinished response easy to mistake for a cancelled request. A user closes the tab, loses the connection, or presses stop; the provider...

Streaming responses and abandoned generations

Updated September 9, 2026 · first published September 9, 2026

Streaming makes an AI product feel fast, but it also makes an unfinished response easy to mistake for a cancelled request. A user closes the tab, loses the connection, or presses stop; the provider may continue generating until the request is cancelled or the output reaches its limit. The visible response is shorter, but the bill is not necessarily shorter.

The abandoned tail is a real cost

Track three events separately: time to first token, tokens delivered to the client, and tokens generated upstream. The gap between delivered and generated output is the abandoned tail. It represents work the user never received, and it can be large in long-form answers, coding agents, and tool-using workflows.

Do not measure this only as a percentage of requests. A small number of abandoned, high-token generations can dominate the waste. Report abandoned output tokens and their cost by feature, model, device, connection type, and cancellation reason.

Cancellation has to travel upstream

Stopping the browser renderer is not the same as stopping the model request. Propagate the client disconnect or an explicit stop action through your gateway to the provider request. Set a short grace period for reconnects, then cancel. For agent workflows, cancel the active generation and any queued tool work that no longer has a live parent task.

Keep an idempotent request identifier so a cancellation race cannot produce duplicate accounting. Record whether the provider confirmed cancellation; otherwise classify the remaining tokens as uncertain spend instead of silently calling them savings.

Use a better KPI

Cost per request is misleading when many responses are abandoned. Use cost per delivered response and cost per successful task, then show the abandonment rate beside both. A product change that increases time to first token may increase abandonment even when total tokens per request stay flat.

Practical controls

Emit a cancellation event with request, feature, model, and token counters.
Cancel upstream work on disconnect and explicit stop.
Cap maximum output for interactive workloads.
Compare generated, delivered, and billed tokens during invoice reconciliation.
Alert when abandoned-token cost rises faster than successful-task volume.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research