The data wall is the forecast that high-quality public text will be fully consumed by frontier training runs on the order of 2027–2028, well before the compute and power constraints discussed in Scaling Laws and Epoch AI — Can AI Scaling Continue Through 2030? bind. The forecast is most prominently associated with Villalobos et al. at Epoch AI, which identifies data as one of four physical scaling bottlenecks alongside compute, power, and latency.
The quantitative claim
Epoch's headline estimates (Villalobos, Sevilla, Besiroglu et al.) place the total stock of indexable high-quality public text on the order of ~10^14 tokens, with substantial uncertainty bands. Frontier training runs in 2024–2025 consumed roughly ~10^13 tokens, rising superlinearly with model size under Chinchilla-compliant recipes. On a continuation of current scaling, Epoch projects exhaustion of high-quality public text in 2027–2028, and earlier if overtraining ratios keep rising as they have for Llama 3, Qwen3, and DeepSeek-V3.
"High-quality" here means text that survives heuristic and classifier filtering: books, curated articles, technical documentation, and high-karma forum content. Low-quality web text, such as AI-generated SEO, boilerplate, and machine translations, exceeds this stock by orders of magnitude but adds little training signal and may degrade results. See Epoch AI — Can AI Scaling Continue Through 2030? for the four-bottleneck framing.
Quality decay
Common Crawl, the primary public web scrape underlying most open-source training corpora, has observably degraded since 2022. Raw text continues to grow while unique-content growth diverges from it; boilerplate templates and mirror sites have become heavier; and LLM output increasingly appears in scraped text, creating distribution-shift and Model Collapse concerns if iterated naively.
The release of ChatGPT in November 2022 is a widely cited inflection point: text crawled after that date is statistically contaminated with LLM-generated content. Some frontier labs now effectively freeze pretraining corpora to pre-2022 snapshots to avoid ingesting model output, at the cost of losing access to newer knowledge.
The pre-2022 preference has produced a physical-book procurement market. 404 Media reported in July 2026 that book-data broker ISBNdb brokers bulk purchases of printed books for AI training data — orders ranging from 1,000 to one million books, marketed as pre-2022 print stock "structurally guaranteed" to be free of AI-generated contamination, conducted under strict NDAs and with the caution to clients that "the optics problem is real." Booksellers described a historic spike in bulk orders beginning in April 2026; the reporting noted that book publishers had sued Google days earlier over Gemini's training on copyrighted books (see Hachette et al. v. Google (Gemini training data)) (Source: 404media.co). See AI Content Saturation (\"AI Slop\").
Access closure
The data wall is not only an absolute-supply problem; legal and contractual access to text is also closing, so the effective legally-usable corpus is smaller than the technically-scrapeable corpus and shrinking faster. Several mechanisms drive this:
- Paywalls and login gates — news publishers including NYT, Reuters, and Axel Springer have both sued and licensed; OpenAI and others have signed nine-figure content deals.
- robots.txt enforcement — Cloudflare's 2024 AI bot blocking and robots.txt expansion shifted millions of sites to explicit disallow.
- Platform licensing exclusives — Reddit, Stack Overflow, and X data are no longer freely crawlable. Licensing relationships themselves are unstable: reporting on July 22, 2026 that Reddit might cut Google's access to its content for AI training sent Reddit shares down 8.3% (Source: cnbc.com).
- Anti-scraping litigation — CFAA theories plus contract claims, including the ongoing NYT v. OpenAI case.
- EU TDM opt-outs — the EU AI Act (Regulation 2024/1689) and copyright-directive text-and-data-mining carve-outs let rights-holders opt out in a machine-readable form, which most major publishers have now exercised.
Responses
Frontier labs have pursued several approaches to the projected shortfall. The most prominent is a shift toward model-generated training data: rejection sampling, best-of-N filtered outputs, teacher-student distillation, and curriculum generation. Qwen3, DeepSeek-V3, and o-series reasoning post-training all rely heavily on synthetic data (see Distillation).
Multimodal expansion offers a second route: video and image corpora are much larger than text and largely un-tapped for pretraining-style regimes, which converts the text wall into a partially soluble problem. A related approach is to shift budget toward longer and more expensive post-training, including RL, RLHF, and reasoning post-training, as the o-series, Claude, and Gemini 3 generation already do; this partially decouples capability gains from pretraining-token supply.
Labs have also turned to direct commercial licensing with publishers, archives such as JSTOR and Elsevier, code forges, and video platforms. This works for well-capitalized incumbents and widens the moat against open-source. Finally, retrieval and tool use route around parametric-knowledge limits with RAG and web tools at inference, shifting load from training data to inference compute.
Model-collapse risk
The most-discussed hazard with synthetic data is that if subsequent generations train on prior generations' output, distributions narrow, tail behaviors vanish, and minority modes disappear. Empirical studies (Shumailov et al. 2024; Dohmatob et al.) show degradation under naive iteration. In practice, frontier labs apply heavy filtering, mix synthetic with real data, and use human-preferred samples, which slows but does not eliminate the risk. See Model Collapse for the mechanism and mitigation debate.
Debates and positions
Whether the wall is real or whether "quality" simply redefines as recipes improve is contested. Some Epoch skeptics, and some OpenAI and Anthropic researchers informally, argue that recipe improvements such as curriculum design, filtering, and reasoning post-training push the effective wall decades into the future. A second contested point is whether synthetic data counts as "the same data": if synthetic samples are conditioned on novel prompts or tools they may add genuine signal, but if not they are distributional rearrangements, and the empirical record is mixed. A third open dispute is how much multimodal data helps with text-task capability, since transfer from video and image to text reasoning is debated.
Relationships
- depends-on: Scaling Laws, Compute Governance
- supports: Epoch AI — Can AI Scaling Continue Through 2030? — one of four physical scaling bottlenecks
- related: Model Collapse, Distillation, Open-Source AI / Open-Weight Models, AI Copyright Litigation — Analysis
See also
- Epoch AI — Can AI Scaling Continue Through 2030? — four-bottleneck analysis.
- (Source: epochai.substack.com) — compute/data gap framing.
- Situational Awareness: One-Year-Later Retrospectives — Aschenbrenner's "unhobbling" frame on post-training.