The Scaling Era: An Oral History of AI, 2019–2025 is a 2025 book by Dwarkesh Patel with Gavin Leech, published by Stripe Press. Its first chapter, "Scaling," subtitled "Why is bigger better?", collects firsthand accounts from researchers and leaders at frontier AI labs, along with independent researchers, about how the scaling hypothesis became the dominant paradigm in AI development.
The book is built from curated excerpts of interviews on the Dwarkesh Podcast. Its publisher's description names Dario Amodei, Demis Hassabis, Ilya Sutskever, Eliezer Yudkowsky and Mark Zuckerberg among the interviewees, and states that the book includes more than 170 definitions and visualizations, classic essays on the theme, and previously unpublished interviews. The excerpted chapter opens with a primer glossary of machine-learning terms and a 1997 epigraph from Hans Moravec.
Summary
The chapter documents how the scaling hypothesis — the idea that making neural networks bigger, with more data and compute, would yield better intelligence — moved from a minority view to the dominant paradigm in AI development. It assembles firsthand accounts from researchers and leaders at frontier AI labs covering the 2019–2025 period.
The chapter is organized as thirteen numbered exchanges between Patel and his interviewees, framed by Patel's own introduction and annotated throughout with margin definitions and footnotes. Interviewees in the chapter are Dario Amodei (CEO of Anthropic), Jared Kaplan (Anthropic cofounder), Ilya Sutskever (cofounder of Safe Superintelligence Inc.), Carl Shulman (independent adviser to Open Philanthropy), Shane Legg (cofounder and chief AGI scientist at Google DeepMind), Sholto Douglas (reinforcement-learning infrastructure lead at Anthropic), Trenton Bricken (interpretability researcher at Anthropic), Leopold Aschenbrenner (cofounder of Situational Awareness LP), Demis Hassabis (cofounder and CEO of Google DeepMind), and Gwern Branwen (freelance writer and researcher).
Patel's framing
Patel's introduction attributes the jump in capability from GPT-2 (2019) to GPT-3.5 mainly to scaling: researchers used roughly the same design as GPT-2 but built a much larger version trained on much more data. A footnote records reports that ChatGPT reached more than 100 million monthly active users within two months of launch, against nine months for TikTok. Patel states that the compute needed to train a leading model is 10 billion times higher than in 2010, and that frontier training compute doubled every six months over that period, four times faster than Moore's law predicts. He writes that OpenAI took a $4 million risk in training GPT-3 because it had found neural scaling laws, curves predicting how much models improve as resources increase, and that their predictive record inspired confidence in $100 million training runs (Scaling Laws). A footnote cites a 2024 Epoch AI estimate that two-thirds of gains come from increasing data and compute rather than from better training algorithms, and a figure reproduces a scaling-law plot from the compute-optimal training paper (Training Compute-Optimal Large Language Models (Chinchilla)) in which loss falls smoothly over a 100,000x increase in training compute.
Patel defines the scaling hypothesis as the idea that this resource-intensive strategy is all it will take to build human-level AI, and possibly systems that surpass it. A footnote notes that the term is used inconsistently: it first referred to increasing dense parameters, then shifted to training compute, and at the time of writing referred to the combined compute used for synthetic-data generation, pretraining, post-training and inference. The introduction also describes a second form, inference scaling (also called test-time compute scaling), in which models are trained to think longer or explicit algorithms explore multiple paths. Patel calls the period since 2016 the scaling era; a footnote cites 2007 and 2008 models of 300 billion and 10 billion parameters as earlier large models, and an Epoch AI regression finding that a separate trend of models breaks off the main trend between 2015 and 2016. Another footnote quotes Amodei on the In Good Company podcast in July 2024 saying training runs then cost $100 million, that models in training were closer to $1 billion, and that $10 or $100 billion runs could arrive in 2025, 2026 or 2027.
Why scaling works
Amodei tells Patel that there is still no satisfying explanation for why scaling works and that it is largely an empirical fact. He offers a tentative analogy to power-law correlations in physics, in which each added order of magnitude captures more of a long-tailed distribution, but says it is unclear why performance scales so smoothly with parameters and data. He distinguishes the loss, which is predictable sometimes to several significant figures, from specific abilities such as arithmetic or coding, which he says are hard to predict and sometimes appear abruptly, comparing the difference to predicting weather averages versus the weather on a given day. He recalls Sutskever telling him early on that the models "just want to learn."
Kaplan describes scaling as the observation that performance trends stay predictable over many orders of magnitude as parameters, data or training compute increase, and says the pattern extends beyond language models to multimodal systems. He sketches a hypothesis in which neural networks fit a curve to a data manifold of some intrinsic dimensionality, so that adding parameters divides the manifold into finer pieces and yields power-law improvement, while cautioning that the existence of such a manifold is itself an assumption. Sutskever says a scaling law describes next-word prediction accuracy and that linking it to reasoning capability is a separate problem; he believes a link exists but calls it complicated, and says reasoning tokens can help.
Evolutionary and biological arguments
Carl Shulman argues that evolution supplies an upper bound on the search needed to produce intelligence, since brute-force trial and error was sufficient. Citing neuroscientist Suzana Herculano-Houzel's work describing the human brain as a scaled-up primate brain, he attributes much of human distinctiveness to a larger brain (more than three times the size of a chimpanzee's) and a longer childhood, which he compares to a bigger model trained for longer. He says capabilities that skeptics identified as fundamental limits, such as Winograd schemas and catastrophic forgetting, have repeatedly gone away through scaling. He explains small animal brains by high mortality, calculating that for an animal with a 50 percent chance of dying every three months, extending learning from three to 30 months cuts the expected benefit by a factor of 1,024, and cites economist Michael Kremer's work finding faster technological progress in larger connected populations, with Tasmania losing technologies such as some fishing techniques. Patel concludes from the exchange that the selective pressures limiting intelligence in animals do not act on neural networks. In the same section Patel frames the stakes: if the scale-up works, AGI could arrive within 10 years; if not, growth of about 2 percent a year would mean decades before a $10 trillion model could be trained (AGI Timelines).
Search, architecture and data
Shane Legg describes foundation models as world models of a kind but argues that creative problem solving requires search, citing AlphaGo's Move 37, and says current language models mimic their training data and will not step beyond it until they incorporate powerful search. He says gaps such as episodic memory will need architectural solutions rather than scaling alone, since the brain separates rapid learning of specifics from slow learning of generalities. Hassabis says how far scaling has gone surprised almost everyone, including its early proponents, and that he would once have said an algorithmic breakthrough was needed for models to form concepts and abstractions. He describes Google DeepMind's effort as roughly half scaling and half inventing new architectures and algorithms, calls it an empirical question whether scaling will hit a wall, and says he is very optimistic that synthetic data and self-play can overcome data bottlenecks, while stating that a new science of data curation is needed (Synthetic Data / Model-Generated Training Data, Training Data Walls). He also points to transfer from coding improvements to general reasoning, and to grounding supplied through human raters in RLHF and through multimodal training. A margin note states that most people interviewed in the book disagree that the supply of high-quality training data is exhausted.
Reliability, agents and test-time compute
Sholto Douglas argues that context length has not been the main obstacle to agents; he attributes it to insufficient reliability in chaining subtasks, and says each additional order of magnitude of scale adds "nines of reliability" (Agentic AI). He calls the jump from GPT-3.5 to GPT-4 large and expects very smart, reliable agents over the next couple of orders of magnitude. Trenton Bricken notes reports that GPT-4 has around 1 trillion parameters against 30 to 300 trillion synapses in the human brain, and says machine-learning research is so empirical that the field is performing a greedy evolutionary search over architectures.
Aschenbrenner describes a "test-time compute overhang": GPT-4's few hundred tokens of chain-of-thought reasoning are, in his comparison, equivalent to three minutes of human thought, and enabling millions of tokens per problem would be a 10,000x increase (Reasoning Models and Chain-of-Thought). He cites AlphaGo research indicating that four orders of magnitude more test-time compute is comparable to a model 3.5 orders of magnitude larger, and describes "unhobbling" (a term he coined, per the chapter's margin note) as a second path to agents alongside scaling. He cites the InstructGPT paper as finding RLHF equivalent to a 100-fold increase in model size in human evaluators' preference ratings (Training Language Models to Follow Instructions with Human Feedback (InstructGPT)), estimates that unlocking this capability will take between six months and three years, and predicts that scaling alone will deliver another preschool-to-high-school-sized jump on top of GPT-4 by 2027 or 2028, with models becoming more like agents or "drop-in remote workers" (Situational Awareness: The Decade Ahead). An editorial footnote states that OpenAI's o1 model, released in September 2024, supports the general claim that LLMs would become able to search and perform chain-of-thought reasoning (OpenAI o-series (o1 → o4-mini)).
Amodei calls the data inefficiency of models one of the remaining mysteries: by his estimate models are two to three orders of magnitude smaller than the human brain but trained on three to four orders of magnitude more data, since a human sees hundreds of millions of words by age 18 against trillions for models. He says he has become skeptical of biological analogies, that he does not think the discrepancy matters if scaling continues, and that model abilities do not seem far from human abilities.
Intellectual history
Gwern Branwen recounts reading Hans Moravec and Ray Kurzweil's connectionist argument in the mid-2000s and initially regarding it as magical thinking, then updating as DanNet, AlexNet and steadily larger datasets and training runs appeared. He describes GPT-3 as the decisive test of scaling and says critics had overlooked earlier results, including a 2017 Baidu paper on scaling laws, the use of Bayesian hyperparameter optimization in developing AlphaZero, and BigGAN's scaling to 300 million images. He argues that the research literature understates the role of compute, trial and error, and serendipity, and cites residual networks as an idea first published in 1988 that could not be demonstrated until 2015 for lack of compute.
Relationships
The scaling hypothesis documented in the chapter draws its intellectual foundation from Richard Sutton's essay The Bitter Lesson. The scaling paradigm described here drove the efficiency gains measured by Epoch AI, for which AI Software Progress provides historical context. The scaling era also serves as the empirical foundation for Dario Amodei's predictions about rapid AI progress in Compressed 21st Century. The chapter includes Dario Amodei's recollection of his work on GPT-2 and GPT-3 at OpenAI before he founded Anthropic.
- depends-on: The Bitter Lesson — Patel's questions and footnotes cite Sutton's essay, which is reprinted in the book's appendix
- supports: Scaling Laws, Scaling Laws for Neural Language Models, Training Compute-Optimal Large Language Models (Chinchilla) — firsthand accounts of the scaling-law findings and a reproduced compute-optimal scaling figure
- related: Training Data Walls, Synthetic Data / Model-Generated Training Data, Reasoning Models and Chain-of-Thought, AGI Timelines, Situational Awareness: The Decade Ahead, Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
- related: Dwarkesh Patel, Jared Kaplan, Ilya Sutskever, Shane Legg, Demis Hassabis, Leopold Aschenbrenner
Provenance
Dwarkesh Patel with Gavin Leech, Stripe Press, 2025. PDF converted to markdown with images on 2026-04-13. The raw file covers the book's front matter, primer glossary and Chapter 1 only; the appendix essays it references (Sutton's "The Bitter Lesson", Nostalgebraist's account of scaling predictability, and Branwen's "Scaling Hypothesis" post) are not included. Publisher page: press.stripe.com.