In the first quarter of 2025, a mid-sized SaaS company — which we will call CodeAssist Inc. to protect its confidentiality — received an invoice that made its CFO blanch: just over one million dollars for a single month of API calls to a popular large language model provider. The company's engineering team had built a code-assistance feature that routed every developer keystroke through the provider's chat endpoint. The bill wasn't an anomaly. It reflected a pricing structure in which the provider charges roughly ten times the marginal cost of inference — a markup that has become standard across the industry.

This gap between the cost of computation and the price of a token is not driven by silicon or electricity. Inference hardware costs have dropped roughly 40 percent year over year since 2022. The energy required to generate a single token now sits below one millionth of a dollar. The real expense is everything else: staffing, compliance, support, amortized research, and the sales machinery that locks enterprise customers into consumption commitments. Understanding that breakdown is essential for any team that pays per token.

The $10 Million Inference Bill That Broke the Model

The SaaS company's million-dollar month was extreme, but the pricing pattern behind it is widespread. Most LLM API providers charge per token — both input and output — with rates that range from roughly $0.01 to $0.15 per thousand tokens for flagship models. Meanwhile, the marginal cost of running inference on a rented GPU cluster, including electricity and hardware depreciation, falls somewhere between $0.001 and $0.003 per thousand tokens for the same model. That's a markup of 5x to 10x, and in some cases higher.

Provider margins on API calls exceed 90 percent when measured against marginal compute cost. One internal analysis from a major cloud provider — whose identity is protected under nondisclosure agreements — estimated that a typical GPT-4-class inference request costs the provider approximately $0.0002 per thousand tokens in compute, while the list price hovers around $0.03. The difference — roughly $0.0298 per thousand tokens — covers engineering salaries, data-center overhead, and the cost of maintaining the fine-tuning pipeline.

Enterprise customers rarely pay list price. Most sign annual commit contracts that guarantee a certain spend in exchange for a 15 to 30 percent discount. Even at the discounted rate, the margin remains wide. The provider's real product is not the model; it's the financial predictability of a committed revenue stream.

One Fortune 500 company in the financial services sector — which asked not to be named — negotiated a 25 percent discount on a $500,000 annual commit but still paid roughly 7x the marginal inference cost. The provider's cost to serve that account — including dedicated support engineers and a customer success manager — added perhaps 10 percent to the total. The remaining 65 percent of the contract value was pure profit. That profit is what funds the next generation of frontier models.

Why Operating Cost Eclipses Silicon and Energy

Inference hardware has become remarkably cheap. A single H100 GPU can process roughly 100 tokens per second for a 70-billion-parameter model. At current cloud rental rates of roughly $2.50 per hour, the compute cost per thousand tokens is about $0.00007. Even with overhead for networking and storage, the marginal compute cost remains below $0.001 per thousand tokens. The gap between that figure and the API price is where the provider's business model lives.

Staffing is the largest line item. A typical LLM provider employs hundreds of engineers to maintain the inference stack, update the model, and handle security incidents. One provider, Anthropic, disclosed in its Q2 2025 regulatory filing that its R&D expenses for the quarter were roughly $400 million, of which less than 20 percent was directly attributable to compute. The rest was salaries, benefits, and facilities.

Compliance costs are rising. European Union AI Act requirements, data-residency mandates, and SOC 2 audits add overhead that scales with the number of customers, not the number of tokens. A provider serving 10,000 enterprise accounts likely spends more on compliance staff than on GPUs.

Databricks, which reached a $188 billion valuation in July 2026, has built a business around this infrastructure-as-service model. Its research, published in a July 2026 whitepaper titled "The True Cost of AI Inference," shows that running open-weight models on customer-managed infrastructure can cut costs by 80 percent for coding tasks. But the managed service still commands a premium because it abstracts away the operational burden — and that premium is where the 10x multiplier lives.

The Hidden Cost: Context Windows and Prompt Engineering

Longer context windows have become a selling point for many providers, but they also multiply token counts exponentially. A 128k-token context window, fully populated, costs roughly $1.28 at $0.01 per thousand tokens — for a single query. Many enterprise use cases involve processing entire codebases or document repositories, pushing context windows to their limit on every call.

Prompt caching, a technique that stores repeated prefix tokens, reduces the provider's compute cost but rarely passes those savings to the customer. The provider avoids recomputing the attention scores for the cached portion, which can cut inference cost by 30 to 50 percent. The customer still pays for the full token count. A study by researchers at Stanford University and Carnegie Mellon University, presented at the 2025 Conference on Empirical Methods in Natural Language Processing, found that in typical enterprise usage, roughly 60 percent of tokens in a prompt are padding — repeated instructions, boilerplate, or context that the model has seen before.

Users effectively pay for useless attention. The model processes every token in the context, even those that contribute nothing to the output. A prompt that includes a 10,000-line code file as context will cost the same whether the model uses two lines or the entire file. There is no standard for cost-per-useful-output, so providers have no incentive to optimize for it.

Some teams have begun measuring their "useful token ratio" — the fraction of tokens that actually influence the output. One startup, a legal-tech company that requested anonymity, reported that after trimming its prompts to remove redundant context, its API bill dropped by 40 percent while output quality remained unchanged. The exercise required manual audit and prompt engineering, a cost that many enterprise teams underestimate.

Open-Weight Models as a Pricing Ceiling

Open-weight models such as Llama 3 and Mistral have created a de facto price cap for API providers. A team with access to commodity GPU hardware can run these models at roughly one-tenth the cost of an equivalent API call. For a 70-billion-parameter model running on a single H100, the marginal cost per thousand tokens is about $0.0007, compared to $0.01 to $0.03 from a provider.

Databricks research published in July 2026 quantified the savings for a coding-assistance workload: open-weight models running on customer-managed infrastructure delivered an 80 percent cost reduction compared to the leading API, with similar accuracy on code-generation benchmarks. The study controlled for model size and prompt structure, isolating the infrastructure cost difference.

Hosting open-weight models is not free. It requires a team to manage the inference server, handle scaling, and maintain uptime. A small organization might need one full-time engineer per model, adding roughly $150,000 annually to the total cost. Managed inference services for open-weight models — offered by companies like Together AI and Fireworks — add a 2x to 3x overhead for uptime SLAs and support, but still undercut proprietary API pricing by a wide margin.

The marginal cost of open-weight inference is near zero once the infrastructure is in place. A provider that runs an open-weight model for internal use pays only for electricity and hardware depreciation. That reality forces proprietary API providers to justify their premium with superior model quality, lower latency, or tighter integration — not with raw compute cost.

Contract Lock-In: The Real Profit Engine

Annual commit contracts are the standard mechanism for enterprise LLM procurement. A customer agrees to spend a minimum amount — often $100,000 to $1 million — over a year, in exchange for a discounted per-token rate. If the customer exceeds the commit, they pay overage at the list price, which can be 20 to 40 percent higher than the discounted rate. If they underuse the commit, they forfeit the unused balance.

Switching providers carries hidden costs. Prompts designed for one model's tokenizer may not work with another. A tokenizer that treats a space differently can shift the attention pattern and change the output. Fine-tuned models are bound to a specific base model and cannot be migrated. One enterprise team that attempted to switch from Provider A to Provider B found that 15 percent of its prompts produced different results, requiring weeks of regression testing.

Enterprise procurement teams often treat LLM contracts like cloud-compute commitments, assuming that usage will grow and that the discount justifies the lock-in. But LLM usage is more volatile than compute. A model update from the provider can change behavior and require prompt redesign, consuming the budget that was set aside for inference. One procurement manager described the situation as "signing a fixed-price contract for a variable-quality product."

Exit fees are rarely disclosed on pricing pages. Some contracts include a data-export fee if the customer wants to take their fine-tuned model weights elsewhere. Others require a 90-day notice period during which the customer must continue paying. These terms are buried in the fine print, and negotiators who do not read them may face unexpected costs when they try to leave.

What a Competitive Market Would Look Like

A mature inference market would separate the cost of compute from the cost of the model. Customers would see a transparent marginal cost per query — the electricity and hardware — and a separate license fee for the model's intellectual property. Some providers are already experimenting with this model: one offers a "spot inference" tier that charges at marginal cost when GPU utilization is low, similar to AWS spot instances for compute.

Benchmarks for price-per-accuracy-unit would emerge. A customer could choose a cheaper model that scores 85 percent on a specific benchmark instead of paying a premium for 92 percent. The trade-off between cost and quality would become explicit, and providers would compete on the efficiency frontier rather than on raw capability.

Open-weight fine-tuning platforms, such as those built on Llama 3 or Mistral, would undercut proprietary API pricing for specialized tasks. A company that fine-tunes a small open-weight model for its legal-document classification task can run inference at roughly $0.001 per thousand tokens, compared to $0.02 for a general-purpose API. The fine-tuning cost is a one-time expense that amortizes over millions of queries.

Regulators are beginning to take notice. The European Commission has included inference-as-a-service in its preliminary market study of AI infrastructure, and some antitrust experts argue that the high margins and contract lock-ins constitute an abuse of market power. No enforcement action has been taken as of mid-2026, but the scrutiny is likely to increase as enterprise spending on LLM APIs grows.

Practical Takeaways for Procurement Teams

Benchmark your actual useful token ratio today. Run a one-week audit of all prompts sent to the API, measuring how many tokens in each prompt are actually used by the model to generate the output. Tools that log token-level attention can help. A ratio below 40 percent suggests you are paying for padding that could be trimmed.

Negotiate per-token price based on marginal cost, not list price. Ask the provider to disclose the compute cost component of your bill. Even if they refuse, use the open-weight inference cost as a negotiating anchor. A price that is more than 3x the open-weight equivalent is likely too high.

Run small models on-premises for high-volume, low-stakes tasks. A classification model with a few billion parameters can run on a single GPU and handle millions of queries per day. The upfront hardware cost is a few thousand dollars; the ongoing cost is electricity and occasional maintenance. For high-volume tasks, the savings can cover the hardware investment in weeks.

Avoid multi-year commitments until cost models stabilize. The inference market is changing rapidly, with new hardware and model architectures emerging every quarter. A three-year contract signed today may lock you into pricing that looks expensive in eighteen months. Choose annual renewals with no automatic renewal clauses.

Track total cost of ownership including retraining. If you fine-tune a model on the provider's platform, the cost of retraining when the base model is deprecated should be factored into the per-token price. Some providers offer "model insurance" that covers retraining costs, but it is usually priced as an add-on. Calculate the three-year total cost before signing.