A 2022 research paper by Hoffmann, Borgeaud, Mensch, and colleagues at Google DeepMind that revised the earlier Kaplan scaling laws. It argues that, for a given training-compute budget, optimal performance requires far more training data relative to model parameters than previously assumed.
Summary of findings
The paper finds that for compute-optimal training, model size and the amount of training data should be scaled roughly equally. By this account, earlier models such as GPT-3 and Gopher were over-parameterized relative to the amount of data they were trained on. To demonstrate the point, the authors trained a 70B-parameter model named "Chinchilla" on four times more data than the 280B-parameter Gopher; Chinchilla outperformed Gopher on nearly every benchmark despite being smaller. The resulting reallocation toward more data and smaller models is referred to as the "Chinchilla rebalancing."
Key claims
- For compute-optimal training, model size and training data should be scaled roughly equally.
- Previous models, including GPT-3 and Gopher, were over-parameterized relative to their training data.
- The 70B-parameter Chinchilla model, trained on 4× more data than the 280B-parameter Gopher, outperformed Gopher on nearly every benchmark.
Reception and influence
The Chinchilla revision is widely treated as the most consequential update to the original Kaplan scaling laws. Epoch AI identifies the Chinchilla rebalancing as one of just two scale-dependent innovations accounting for the bulk of measured AI software progress. The rapid, across-the-board adoption of the result by frontier labs after publication is cited as an example of the fast-follow problem, in which algorithmic knowledge spreads quickly once disclosed.
Provenance
Source: Hoffmann, Borgeaud, Mensch, et al. (Google DeepMind), 2022. PDF converted to markdown on 2026-04-13.