Inference Economics and Token Pricing

Inference economics is the cost side of AI deployment: the cost of running a model once it has been trained, and how that cost evolves. Across 2023–2026 the pattern has been rapid deflation in per-token prices alongside growth in aggregate spend, a dynamic often described as a Jevons paradox. Over the same period inference compute has grown relative to training compute, reaching roughly parity in aggregate.

Core metrics

Inference cost and performance are measured along several units. Cost per million input tokens ($/M input) and cost per million output tokens ($/M output) are the primary price metrics; output is typically 3–5× input because of autoregressive generation. Throughput is measured in tokens per second, and latency in part by time-to-first-token (TTFT). Cost per task is increasingly the relevant unit as reasoning and agentic workflows blur the boundary set by raw token counts. Aggregate volume figures illustrate the scale of serving demand: China's National Data Administration reported daily national token consumption of 140 trillion in March 2026, up from 100 trillion in December 2025 and 100 billion in early 2024 (Source: scmp.com).

Price deflation

Per-token prices have fallen roughly 90% or more from 2023 peaks at comparable capability tiers. Open-weights models hosted by third parties (Llama, DeepSeek, Qwen) have consistently undercut closed models by 3–10×.

Snapshot — representative frontier/tier pricing ($/M input / $/M output)

Deflation has been driven by several factors. Hardware generation gains have lowered cost per token across the Nvidia H100 → H200 → B200 → GB200 progression. Mixture of Experts (Mixture of Experts (MoE)) reduces active-parameter inference cost. KV cache optimizations, including paged attention and shared prefix caching, reduce repeated work. Speculative decoding, in which a draft model proposes tokens that the target model verifies, accelerates generation by 2–4×. Batching lets inference servers pack many requests together to amortize fixed costs. Quantization (FP8, INT4) shrinks memory bandwidth bottlenecks. Distillation (Distillation) yields smaller models at a given capability tier.

Analyst Benedict Evans argued in a July 9, 2026 essay that every visible market dynamic points to frontier models becoming low-margin commodity infrastructure as the token supply crunch eases. He cited reported inference gross margins of 40–50 percent — noting these include server depreciation on an unknown asset life but exclude training costs, which he says currently exceed revenue — and compared token-selling to mobile data, a trillion-dollar-revenue industry with $200 billion of annual capex whose value was captured further up the stack (Ways to think about token pricing (Benedict Evans, July 2026)). Evans framed the open variables as four questions: how much demand is absorbed by cheaper models that are "good 'enough'" rather than the frontier; whether the frontier keeps improving fast enough to outrun efficiency and capacity gains; whether frontier competition stays fierce or consolidates; and how much of the value from high-end use cases the model itself captures rather than the tooling, proprietary data, and go-to-market wrapped around it. His asymmetry argument is that commoditization is the default and every alternative outcome "requires something to change." The July 2026 pricing round was read in similar terms: Bloomberg analysis of the week's near-simultaneous releases (GPT-5.6 Sol at $5/$30 per million tokens, Muse Spark 1.1 at $1.25/$4.25, Grok 4.5 at $2/$6) described cost efficiency rather than raw capability as the models' biggest immediate selling point (Source: bloomberg.com). See Coding AI Market Map.

The commoditization reading sharpened after Kimi K3's July 16 release. Alex Kantrowitz argued in a July 17, 2026 analysis that Muse Spark 1.1, Grok 4.5, and Kimi K3 — all released within eight days — were setting off an AI price war just as OpenAI's and Anthropic's IPO windows opened, quoting Atreides Management's Gavin Baker that lower model-layer margins are "good for every other AI layer" (Source: bigtechnology.com). A same-day Wall Street Journal analysis likewise argued that intelligence "too cheap to meter" threatens the leading labs while creating opportunity for competitors (Source: wsj.com). Chris Zeoli's July 18, 2026 analysis of the pending Kimi K3 weights release quantified the same mechanism: open frontier weights shift margin from closed labs, whose API gross margins he estimates above 80 percent, toward the compute and serving layers at roughly 40–45 percent — a swing he sizes near $90 billion a year against a $250 billion 2030 inference-market forecast (Source: datagravity.dev; exponentialview.co).

Jevons paradox

Cheaper tokens have not reduced aggregate AI spend; they have expanded it. Token volume has grown roughly 10× annually over 2023–2026, while per-token prices have fallen 5–10× over the same period, leaving aggregate inference spend higher. Per Epoch AI analyses, inference compute has reached roughly parity with training compute in aggregate, often cited as a ~1:1 split, a shift from the ~90:10 training-heavy split of 2022.

Reasoning models (see Reasoning Models and Chain-of-Thought) tilt the balance further toward inference by consuming many "thinking" tokens per user-visible answer.

Pricing tiers

Frontier labs have converged on a tiered model family that gives deployers a cost-quality dial and lets labs price across workloads. The flagship tier (Claude Opus, GPT-4.5/5 family, Gemini Pro) offers maximum capability at premium price. The workhorse tier (Claude Sonnet, GPT-4o family, Gemini Flash) sits close to the frontier at a fraction of the cost and absorbs most enterprise volume. The economy tier (Claude Haiku, GPT-4o-mini / nano, Gemini Nano) is priced for high throughput on simple tasks.

API versus chat product unit economics

API access is metered per token, with margin depending on inference efficiency. Chat products (ChatGPT, Claude.ai, Gemini) charge a flat subscription, typically a $20/month consumer tier, where unit economics are dominated by heavy power-users who can consume many dollars of inference for a flat fee. Enterprise chat uses per-seat pricing with usage caps or metered overages.

OpenAI, Anthropic, and Google all run subscription products at negative or thin marginal margin on heavy users, subsidized by light users and by the flywheel of training data. Reasoning-model subscriptions have strained this arrangement, because long reasoning traces are expensive to serve.

The strain has extended to enterprise deployers as providers, including GitHub, shift from flat subscriptions to per-token pricing. A July 2026 404 Media investigation based on obtained internal emails found companies throttling — and in some cases cutting off — employees' access to AI models to stop burning through token budgets (Source: 404media.co). Moody's Ratings analysis circulating July 21, 2026 reached a parallel conclusion from the credit side: spending on AI tokens will keep rising largely to the benefit of AI "suppliers," while adopting companies "foot the bills" and must change their business models to translate the investment into financial gains (Source: insideaipolicy.com).

Alex Kantrowitz argued on July 31, 2026 that the metered-API model may not persist at the top of the range: as the field fills with models of comparable quality, selling access to a model becomes a commodity business, and laboratories may therefore withhold their most capable models from the API and use them to build proprietary products instead. On that reasoning the frontier would become less, not more, commercially accessible as it advances. The page was retrieved as a summary rather than full text, so the argument is recorded at that level of detail (Source: bigtechnology.com).

Key technical mechanisms

Several serving techniques shape inference cost. The KV cache stores attention over prior tokens so each new token only attends over cached keys and values rather than recomputing them; cache size scales with context length and model size. Prefix caching extends this by caching shared system prompts across requests, and a common pricing pattern charges a reduced rate on cache-hit tokens. Speculative decoding uses a small draft model to propose k tokens that the large model verifies in parallel. Continuous batching lets new requests join a running batch mid-flight to keep GPUs utilized. Expert parallelism, used for MoE models, routes tokens to experts that may sit on different GPUs; it is infrastructure-heavy but enables serving of trillion-parameter models.

Policy and safety relevance

Compute governance (Compute Governance) has focused overwhelmingly on training; inference regulation such as throughput caps or deployment licensing is rare and contested. Some safety proposals advocate inference-time monitoring as a control layer. On competitiveness, cheaply hosted open-weights models are a vector of Fast-Follow Problem competitive pressure. Inference is now the larger share of AI energy demand (see AI Environmental Impact). Export controls (Export controls) currently target training chips, while the H20 and B20 chips are optimized for inference, creating a gap. On access and equity, cheap inference broadens access, while expensive inference creates divides between well- and poorly-resourced users.

Buyer-side cost pressure (2026)

By July 24, 2026, enterprise buyers including Uber, Meta, Microsoft, Salesforce and DoorDash had launched AI cost-cutting campaigns after model bills doubled or tripled, or consumed annual budgets within three months. The reported responses were to shift from single-vendor commitments toward mixing lower-priced models — including Chinese-built ones — alongside OpenAI and Anthropic products, to ration employee access, and to steer staff toward cheaper internal alternatives (Source: wsj.com). Moody's Ratings concluded by the same date that spending on AI tokens will keep rising largely to the benefit of AI suppliers, while adopting companies bear the costs and must change their business models to convert the investment into financial gains (Source: insideaipolicy.com). Both readings sit on the demand side of the Jevons question above: falling per-token prices have not held total spend flat for adopters. See Enterprise AI Deployment Gap, AI Bubble vs. Buildout — Synthesis.

Hardware and serving landscape

Frontier inference serving is dominated by Nvidia (GB200, H100/H200), with AMD MI300/MI325 and Google TPUs as the main alternatives. Specialty inference hardware from Cerebras, Groq, and SambaNova targets high throughput on specific model families. Per-token prices have deflated roughly in line with Moore's-law-scale trajectories; whether this continues depends on hardware cadence.

Financial-market infrastructure around compute pricing has begun to form: on July 14, 2026, the prediction-market platform Kalshi launched a forward curve tracking expected AI compute rental costs, combining weekly and monthly event contracts into a GPU rental-price estimate that could underpin futures, options, and swaps for hedging infrastructure costs (Source: bloomberg.com).

GPU rental prices began moving toward an exchange-traded reference in 2026. CME Group and Silicon Data announced on August 11, 2026 that they plan to launch two Compute futures contracts on October 5, 2026, pending regulatory review: Silicon Data H100 Rental Index Futures and Silicon Data B200 Rental Index Futures, each representing a month's worth of rent for the respective Nvidia chip and tracking indexes of hourly GPU rental costs published by Silicon Data (Source: cmegroup.com). These would be the first exchange-listed contracts on GPU rental prices, making the rental rate hedgeable rather than only observable. See Private Credit & AI Infrastructure.

Relationships

• depends-on: Scaling Laws — training-compute trajectory shapes what needs to be served
• related: Mixture of Experts (MoE) — MoE is an inference-efficiency lever
• related: Distillation — main mechanism for cheap-tier model economics
• related: Reasoning Models and Chain-of-Thought — reasoning tilts economics toward inference
• related: AI Environmental Impact, Compute Governance, Semiconductor Supply Chain
• related: Enterprise AI Deployment Gap