Inference Economics and Token Pricing

Inference economics is the cost side of AI deployment: the cost of running a model once it has been trained, and how that cost evolves. Across 2023–2026 the pattern has been rapid deflation in per-token prices alongside growth in aggregate spend, a dynamic often described as a Jevons paradox. Over the same period inference compute has grown relative to training compute, reaching roughly parity in aggregate.

Core metrics

Inference cost and performance are measured along several units. Cost per million input tokens ($/M input) and cost per million output tokens ($/M output) are the primary price metrics; output is typically 3–5× input because of autoregressive generation. Throughput is measured in tokens per second, and latency in part by time-to-first-token (TTFT). Cost per task is increasingly the relevant unit as reasoning and agentic workflows blur the boundary set by raw token counts. Aggregate volume figures illustrate the scale of serving demand: China's National Data Administration reported daily national token consumption of 140 trillion in March 2026, up from 100 trillion in December 2025 and 100 billion in early 2024 (Source: scmp.com).

Price deflation

Per-token prices have fallen roughly 90% or more from 2023 peaks at comparable capability tiers. Open-weights models hosted by third parties (Llama, DeepSeek, Qwen) have consistently undercut closed models by 3–10×.

Snapshot — representative frontier/tier pricing ($/M input / $/M output)

Deflation has been driven by several factors. Hardware generation gains have lowered cost per token across the Nvidia H100 → H200 → B200 → GB200 progression. Mixture of Experts (Mixture of Experts (MoE)) reduces active-parameter inference cost. KV cache optimizations, including paged attention and shared prefix caching, reduce repeated work. Speculative decoding, in which a draft model proposes tokens that the target model verifies, accelerates generation by 2–4×. Batching lets inference servers pack many requests together to amortize fixed costs. Quantization (FP8, INT4) shrinks memory bandwidth bottlenecks. Distillation (Distillation) yields smaller models at a given capability tier.

Analyst Benedict Evans argued in a July 9, 2026 essay that every visible market dynamic points to frontier models becoming low-margin commodity infrastructure as the token supply crunch eases. He cited reported inference gross margins of 40–50 percent — noting these include server depreciation on an unknown asset life but exclude training costs, which he says currently exceed revenue — and compared token-selling to mobile data, a trillion-dollar-revenue industry with $200 billion of annual capex whose value was captured further up the stack (Ways to think about token pricing (Benedict Evans, July 2026)). Evans framed the open variables as four questions: how much demand is absorbed by cheaper models that are "good 'enough'" rather than the frontier; whether the frontier keeps improving fast enough to outrun efficiency and capacity gains; whether frontier competition stays fierce or consolidates; and how much of the value from high-end use cases the model itself captures rather than the tooling, proprietary data, and go-to-market wrapped around it. His asymmetry argument is that commoditization is the default and every alternative outcome "requires something to change." The July 2026 pricing round was read in similar terms: Bloomberg analysis of the week's near-simultaneous releases (GPT-5.6 Sol at $5/$30 per million tokens, Muse Spark 1.1 at $1.25/$4.25, Grok 4.5 at $2/$6) described cost efficiency rather than raw capability as the models' biggest immediate selling point (Source: bloomberg.com). See Coding AI Market Map.

The commoditization reading sharpened after Kimi K3's July 16 release. Alex Kantrowitz argued in a July 17, 2026 analysis that Muse Spark 1.1, Grok 4.5, and Kimi K3 — all released within eight days — were setting off an AI price war just as OpenAI's and Anthropic's IPO windows opened, quoting Atreides Management's Gavin Baker that lower model-layer margins are "good for every other AI layer" (Source: bigtechnology.com). A same-day Wall Street Journal analysis likewise argued that intelligence "too cheap to meter" threatens the leading labs while creating opportunity for competitors (Source: wsj.com). Chris Zeoli's July 18, 2026 analysis of the pending Kimi K3 weights release quantified the same mechanism: open frontier weights shift margin from closed labs, whose API gross margins he estimates above 80 percent, toward the compute and serving layers at roughly 40–45 percent — a swing he sizes near $90 billion a year against a $250 billion 2030 inference-market forecast (Source: datagravity.dev; exponentialview.co).

Jevons paradox

Cheaper tokens have not reduced aggregate AI spend; they have expanded it. Token volume has grown roughly 10× annually over 2023–2026, while per-token prices have fallen 5–10× over the same period, leaving aggregate inference spend higher. Per Epoch AI analyses, inference compute has reached roughly parity with training compute in aggregate, often cited as a ~1:1 split, a shift from the ~90:10 training-heavy split of 2022.

Reasoning models (see Reasoning Models and Chain-of-Thought) tilt the balance further toward inference by consuming many "thinking" tokens per user-visible answer.

Exponential View's June 2026 model estimates the elasticity directly. Fitting Google's average price against volume on a log-log scale gives a trend β of −1.7 with an R² of 0.93, and across providers the magnitude of elasticity is put at 1.2–1.8: every 10% price cut yields 12–18% more tokens, so total token spend still rises. Three executive datapoints anchor the fit — Sundar Pichai reporting Google moving from 9.7 trillion tokens a month to over 480 trillion, a 50x rise against a 97% price decline; Sam Altman's statement that "the cost to use a given level of AI falls about 10x every 12 months, and lower prices lead to much more use," against a 90% price decline; and ByteDance's Tan Dai reporting Doubao daily token usage above 50 trillion, up from 4 trillion in December 2024, against a 50% price decline. The authors caveat that the regression is a time series rather than a cross-section, so price and usage both trend with time and β "may overstate pure price-elasticity" (The State of the AI Economy (Exponential View, June 2026)).

The same report puts global inference volumes above 30 quadrillion tokens a month growing 14x year-on-year (a series that, unlike its revenue series, includes China), and decomposes the joint movement of capability and price: the Epoch Capabilities Index rose from 112 to 158 while the blended price per million tokens fell from $17 to $2 and tokens processed per output token rose from 12 to 36 (The State of the AI Economy (Exponential View, June 2026)).

Cost floor per gigawatt

A capacity-based cost model in the same report derives a price floor from data-center economics rather than from published rate cards. Owning and operating 1 GW of AI capacity is modelled at $7.9 billion a year — 89% capital, 11% operating — against an output of 75–85 quadrillion tokens per gigawatt per year, giving an inference-provider cost of about $0.10 per million tokens for an open-weight model. Earning a 50–75% gross margin on that requires $0.20–$0.40 per million tokens, and a 25% return on investment requires $0.25–$0.50 of customer value. For an illustrative closed-licensed model of Kimi class the cost rises to $0.42, of which $0.32 is a licensing fee set at roughly 25% of Kimi's $1.29 blended selling price, requiring $0.84–$1.68 in price and $1.05–$2.10 in customer value on the same tests. The model is explicitly illustrative: 6,700 GB200 NVL72 systems and 480,000 GPUs, cost of ownership per Epoch AI (May 2026) annualized including cost of capital on a six-year IT life, token output from SemiAnalysis InferenceX at FP4, 8k-in/1k-out, 50 tokens per second per user and 65% utilization, with the range reflecting a 10–25% throughput uplift from speculative decoding (The State of the AI Economy (Exponential View, June 2026)).

Output per unit of physical capacity rises each generation — from smaller models and better serving, from hardware gains across Hopper, Blackwell and Rubin, and from a workload mix shifting toward inference — so revenue per trillion tokens has fallen since its 2023 peak while industry-wide revenue per gigawatt of data-center capacity passed $7 billion per GW (The State of the AI Economy (Exponential View, June 2026)).

Pricing tiers

Frontier labs have converged on a tiered model family that gives deployers a cost-quality dial and lets labs price across workloads. The flagship tier (Claude Opus, GPT-4.5/5 family, Gemini Pro) offers maximum capability at premium price. The workhorse tier (Claude Sonnet, GPT-4o family, Gemini Flash) sits close to the frontier at a fraction of the cost and absorbs most enterprise volume. The economy tier (Claude Haiku, GPT-4o-mini / nano, Gemini Nano) is priced for high throughput on simple tasks.

Outcome-based pricing

A second axis emerged in 2026 alongside the capability tiers: what unit is billed at all. Salesforce began letting customers negotiate Agentforce contracts priced on revenue gained or cost avoided rather than per task, with chief executive Marc Benioff describing the goal on an August 26, 2026 investor call as being able to say "we improve revenue by this much, so give us $2 because we made you $20 or we made you $40." OpenAI has offered some large customers the option of paying only when its AI completes tasks such as customer-support interactions, and Cognition has promised enterprise customers up to $10 million in credits if it fails to deliver engineering results worth at least what they pay. Stripe has published guidelines for outcome-based pricing warning that without explicit attribution rules, customers may dispute whether an outcome belongs to the vendor (Source: theinformation.com). The arrangement moves the risk of an agent's failure from the buyer to the vendor and makes attribution, rather than token count, the contested measurement.

API versus chat product unit economics

API access is metered per token, with margin depending on inference efficiency. Chat products (ChatGPT, Claude.ai, Gemini) charge a flat subscription, typically a $20/month consumer tier, where unit economics are dominated by heavy power-users who can consume many dollars of inference for a flat fee. Enterprise chat uses per-seat pricing with usage caps or metered overages.

OpenAI, Anthropic, and Google all run subscription products at negative or thin marginal margin on heavy users, subsidized by light users and by the flywheel of training data. Reasoning-model subscriptions have strained this arrangement, because long reasoning traces are expensive to serve.

The strain has extended to enterprise deployers as providers, including GitHub, shift from flat subscriptions to per-token pricing. A July 2026 404 Media investigation based on obtained internal emails found companies throttling — and in some cases cutting off — employees' access to AI models to stop burning through token budgets (Source: 404media.co). Moody's Ratings analysis circulating July 21, 2026 reached a parallel conclusion from the credit side: spending on AI tokens will keep rising largely to the benefit of AI "suppliers," while adopting companies "foot the bills" and must change their business models to translate the investment into financial gains (Source: insideaipolicy.com).

Alex Kantrowitz argued on July 31, 2026 that the metered-API model may not persist at the top of the range: as the field fills with models of comparable quality, selling access to a model becomes a commodity business, and laboratories may therefore withhold their most capable models from the API and use them to build proprietary products instead. On that reasoning the frontier would become less, not more, commercially accessible as it advances. The page was retrieved as a summary rather than full text, so the argument is recorded at that level of detail (Source: bigtechnology.com).

Key technical mechanisms

Several serving techniques shape inference cost. The KV cache stores attention over prior tokens so each new token only attends over cached keys and values rather than recomputing them; cache size scales with context length and model size. Prefix caching extends this by caching shared system prompts across requests, and a common pricing pattern charges a reduced rate on cache-hit tokens. Speculative decoding uses a small draft model to propose k tokens that the large model verifies in parallel. Continuous batching lets new requests join a running batch mid-flight to keep GPUs utilized. Expert parallelism, used for MoE models, routes tokens to experts that may sit on different GPUs; it is infrastructure-heavy but enables serving of trillion-parameter models.

Policy and safety relevance

Compute governance (Compute Governance) has focused overwhelmingly on training; inference regulation such as throughput caps or deployment licensing is rare and contested. Some safety proposals advocate inference-time monitoring as a control layer. On competitiveness, cheaply hosted open-weights models are a vector of Fast-Follow Problem competitive pressure. Inference is now the larger share of AI energy demand (see AI Environmental Impact). Export controls (Export controls) currently target training chips, while the H20 and B20 chips are optimized for inference, creating a gap. On access and equity, cheap inference broadens access, while expensive inference creates divides between well- and poorly-resourced users.

Buyer-side cost pressure (2026)

By July 24, 2026, enterprise buyers including Uber, Meta, Microsoft, Salesforce and DoorDash had launched AI cost-cutting campaigns after model bills doubled or tripled, or consumed annual budgets within three months. The reported responses were to shift from single-vendor commitments toward mixing lower-priced models — including Chinese-built ones — alongside OpenAI and Anthropic products, to ration employee access, and to steer staff toward cheaper internal alternatives (Source: wsj.com). Moody's Ratings concluded by the same date that spending on AI tokens will keep rising largely to the benefit of AI suppliers, while adopting companies bear the costs and must change their business models to convert the investment into financial gains (Source: insideaipolicy.com). Both readings sit on the demand side of the Jevons question above: falling per-token prices have not held total spend flat for adopters. See Enterprise AI Deployment Gap, AI Bubble vs. Buildout — Synthesis.

Hardware and serving landscape

Frontier inference serving is dominated by Nvidia (GB200, H100/H200), with AMD MI300/MI325 and Google TPUs as the main alternatives. Specialty inference hardware from Cerebras, Groq, and SambaNova targets high throughput on specific model families. Per-token prices have deflated roughly in line with Moore's-law-scale trajectories; whether this continues depends on hardware cadence.

A model developer's own silicon entered the published comparison in August 2026. OpenAI released measured results for Jalapeño, the inference accelerator it developed with Broadcom, on SemiAnalysis's InferenceX benchmark across GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T: 1.5 to 1.9 times more work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than Nvidia GB200 and GB300 comparison systems. The unit of comparison is the point of the claim — OpenAI states that although performance is sometimes reported per chip, it takes performance per unit of power as the more useful standard, and normalizes every ratio by rated chip power (700 W against 1,200 W and 1,400 W). The economic argument the company attaches is operating leverage rather than price: producing more useful work from the same power lowers the cost of delivering a result, so useful work and revenue can grow faster than the cost to serve (Jalapeño's first results show industry-leading speed and efficiency in AI inference). The figures are a vendor self-report on a public benchmark, not independently reproduced.

Financial-market infrastructure around compute pricing has begun to form: on July 14, 2026, the prediction-market platform Kalshi launched a forward curve tracking expected AI compute rental costs, combining weekly and monthly event contracts into a GPU rental-price estimate that could underpin futures, options, and swaps for hedging infrastructure costs (Source: bloomberg.com).

GPU rental prices began moving toward an exchange-traded reference in 2026. CME Group and Silicon Data announced on August 11, 2026 that they plan to launch two Compute futures contracts on October 5, 2026, pending regulatory review: Silicon Data H100 Rental Index Futures and Silicon Data B200 Rental Index Futures, each representing a month's worth of rent for the respective Nvidia chip and tracking indexes of hourly GPU rental costs published by Silicon Data (Source: cmegroup.com). These would be the first exchange-listed contracts on GPU rental prices, making the rental rate hedgeable rather than only observable. See Private Credit & AI Infrastructure.

Second-hand and retired hardware

A 2026 preprint by Cambridge, Oxford and Quettaflop AI researchers measured the inference economics of the tier below the frontier fleet, reporting one year of production operation of a 128-GPU cluster built entirely from second-hand NVIDIA V100s (DumpsterCluster: From Dumpster Diving to Serving LLaMA-70B on $60 GPUs). Fitting eBay price data to a stretched exponential decay model, the authors put V100 16GB second-hand prices at about $60 by March 2026 — a 99.4% decline from the $10,000 May 2018 launch price — and A100 40GB prices at about $1,900 (84.2% decline), with fitted exponents above 1 in both cases indicating accelerating depreciation. The full 128-GPU pod including peripherals cost about $22,000 as of March 2026, against $600,000 for an eight-GPU B200 system.

Using a custom Rust serving engine implementing device-level pipeline parallelism, the cluster reached about 224,000 tokens per second on a prefill-heavy LLaMA 3.1-8B workload, above the roughly 62,000 TPS the authors measured on a single B200, though on ShareGPT traces it delivered 21,500 TPS against the B200's 44,000. On LLaMA 3.1-70B the cluster reached 1,512 TPS against 37,045 for an eight-GPU B200 pod, a configuration the authors note is impossible on individual V100s at all because the model does not fit in 16 GB. Total cost of ownership reverses by model size: for the 8B model the V100 pod is the cheapest option across all four energy mixes modeled and up to 6× more cost-effective than an H100 system under the China grid mix, while for the 70B model its three-year cost per token is roughly 1.5× that of an H100 system. The authors benchmark both against Together AI's listed prices of $0.10 per million tokens for 8B and $0.88 for 70B, concluding V100-based serving can remain profitable at the higher operating cost. The economics are explicitly pinned to March 2026 second-hand prices, and the authors flag that broad adoption would raise those prices and erode the advantage. The carbon side of the same measurement runs the other way (AI Environmental Impact).

Relationships

• depends-on: Scaling Laws — training-compute trajectory shapes what needs to be served
• related: Mixture of Experts (MoE) — MoE is an inference-efficiency lever
• related: Distillation — main mechanism for cheap-tier model economics
• related: Reasoning Models and Chain-of-Thought — reasoning tilts economics toward inference
• related: AI Environmental Impact, Compute Governance, Semiconductor Supply Chain
• related: Enterprise AI Deployment Gap