"Textbooks Are All You Need" is a June 2023 paper from Microsoft Research by Suriya Gunasekar, Yi Zhang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li, and colleagues. It introduces the phi model family and is the canonical citation for data quality as a substitute for scale.
The claim against scale
The paper situates itself against the scaling picture: performance "improves somewhat predictably as one scales up either the amount of compute or the size of the network," while "the science behind this success remains limited."
Its thesis borrows from Eldan and Li's TinyStories, a synthetic dataset for teaching English to small networks, and extends it: "improving data quality can dramatically change the shape of the scaling laws, potentially allowing to match the performance of large-scale models with much leaner training/models." The stated advance over TinyStories is that "high quality data can even improve the SOTA of large language models, while dramatically reducing the dataset size and training compute." A secondary argument is environmental: "smaller models requiring less training can significantly reduce the environmental cost of LLMs."
See Scaling Laws.
phi-1
| Property | Value |
|---|---|
| Parameters | 1.3B, Transformer-based |
| Training | 4 days on 8 A100s |
| Data | 6B tokens of "textbook quality" web data + 1B tokens of textbooks and exercises synthesized with GPT-3.5 |
| HumanEval pass@1 | 50.6% |
| MBPP | 55.5% |
Two ablation models frame the result. phi-1-base is the model before the finetuning stage on coding exercises; the paper reports "surprising emergent properties" in phi-1 relative to it. phi-1-small, at 350M parameters through the same pipeline, "still achieves 45% on HumanEval" — which is the more striking number, since it holds most of the performance at roughly a quarter of the size.
The task domain is deliberately narrow: writing simple Python functions from their docstrings, evaluated on HumanEval.
Caveats the paper states
Contamination. Section 5 studies "possible contamination of our training data with respect to HumanEval" — necessary because training on GPT-3.5-generated exercises in the target format raises exactly that question.
Withheld method. "We release the model for usage and evaluation by the broader community, but omit some details of the synthetic data generation, for proprietary reasons." The central methodological contribution is therefore not fully reproducible from the paper.
Recursive training. The authors place the work in "the emerging trend of using existing LLMs to synthesize data for the training of new generations of LLMs," and note without resolving it that "there is an ongoing debate about whether such 'recursive training' might lead to narrower scope" — the concern later discussed as model collapse. See Synthetic Data / Model-Generated Training Data.
Benchmark methodology. The evaluation approach is described as "reminiscent of the Sparks of AGI paper," which argued for moving away from static benchmarks; Section 4 discusses alternative benchmarks. See AI Benchmarks and Evaluation.
Relationships
- contradicts (partial): Scaling Laws — argues data quality changes the shape of the scaling curve rather than merely shifting position along it
- supports: Synthetic Data / Model-Generated Training Data — an early demonstration that model-generated training data can improve a frontier result, with the recursive-training concern flagged
- related: Distillation, AI Benchmarks and Evaluation, Inference Economics and Token Pricing