"Scaling Laws for Neural Language Models" is a 2020 paper by Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei (Johns Hopkins University / OpenAI), dated January 23, 2020. It established that language model performance improves as a power law of training compute, dataset size, and model parameters.
The paper was submitted to arXiv on January 23, 2020 as arXiv:2001.08361, in the Machine Learning (cs.LG) and Machine Learning (stat.ML) subject classes, and runs 19 pages with 15 figures (Source: https://arxiv.org/abs/2001.08361). Sam McCandlish is listed as the submitting author (Source: https://arxiv.org/abs/2001.08361).
Summary
The paper reports that language model performance, measured by cross-entropy loss, scales as a power law with each of training compute, dataset size, and the number of model parameters taken individually. The paper characterizes these scaling relationships as predictable, which allows model performance to be forecast before training.
The authors state that they study empirical scaling laws for language-model performance on the cross-entropy loss, and that the loss scales as a power law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude (Source: https://arxiv.org/abs/2001.08361). They report that other architectural details, such as network width or depth, have minimal effects within a wide range, and that simple equations govern the dependence of overfitting on model and dataset size and the dependence of training speed on model size (Source: https://arxiv.org/abs/2001.08361).
Key claims
- Performance (cross-entropy loss) scales as a power law with each of compute, data, and parameters individually, with some trends spanning more than seven orders of magnitude (Source: https://arxiv.org/abs/2001.08361).
- Larger models are more sample-efficient: they reach the same performance with less data. The paper states that larger models are significantly more sample-efficient (Source: https://arxiv.org/abs/2001.08361).
- Architectural details such as network width or depth have minimal effects within a wide range, relative to model size, data, and compute (Source: https://arxiv.org/abs/2001.08361).
- The authors argue that the scaling relationships allow the optimal allocation of a fixed compute budget to be determined, and conclude that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence (Source: https://arxiv.org/abs/2001.08361).
- The scaling laws in the paper favored larger models trained on less data, a recommendation later revised by the Chinchilla paper.
Reception and revision
The recommendation to allocate most of a growing compute budget to model size rather than to training data was revised by Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models" (DeepMind), the paper associated with the Chinchilla model. That work reports that, for compute-optimal training, model size and the number of training tokens should be scaled equally, so that for every doubling of model size the number of training tokens should also be doubled (Source: https://arxiv.org/abs/2203.15556). The authors of that paper state that they trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, and conclude that then-current large language models were significantly undertrained as a consequence of scaling model size while keeping training data roughly constant (Source: https://arxiv.org/abs/2203.15556). They tested the hypothesis by training Chinchilla, a 70-billion-parameter model trained on roughly four times more data than the 280-billion-parameter Gopher under the same compute budget, and report that Chinchilla outperformed Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) across a range of downstream tasks, reaching an average accuracy of 67.5% on the MMLU benchmark (Source: https://arxiv.org/abs/2203.15556). The DeepMind paper is summarized separately on the Chinchilla page.
The Kaplan et al. paper is one of several early-2020s works establishing empirical scaling behavior for neural networks. Secondary surveys place it alongside Hestness et al. (2017) and a 2020 analysis by Henighan, Kaplan, and others that extended scaling-law observations beyond language to other modalities (Source: https://en.wikipedia.org/wiki/Neural_scaling_law). The broader literature frames neural scaling laws as empirical regularities describing how performance varies with model size, dataset size, training cost, and inference compute, generally showing improvement with scale subject to diminishing returns (Source: https://en.wikipedia.org/wiki/Neural_scaling_law).
Relation to other topics
The paper is the original source for the concept summarized on Scaling Laws. The shift from the Kaplan scaling regime to Chinchilla scaling is identified by Epoch AI as one of the two largest single sources of software progress (see AI Software Progress). Co-author Dario Amodei worked on scaling laws at OpenAI before founding Anthropic. Because the paper presents performance as following predictable power laws of compute, it is cited in discussions of Compute Governance, where the inference drawn is that controlling compute constrains capability.
Provenance
PDF converted to markdown on 2026-04-13. Bibliographic and abstract details verified against the arXiv listing for arXiv:2001.08361 (Source: https://arxiv.org/abs/2001.08361).