"A Careful Examination of Large Language Model Performance on Grade School Arithmetic" is a paper by Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati and Summer Yue of Scale AI (Lunati and Yue as equal senior authors), posted to arXiv as 2405.00332 in May 2024 and revised through version 4. It introduces GSM1k, a privately held replica of GSM8K, to measure how far reported GSM8K scores reflect contamination rather than reasoning.
Summary
The authors note that public benchmarks such as GSM8K, MATH, MBPP, HumanEval and SWE-bench may be contaminated because models train on web-scale data that can include items resembling benchmark questions, so that a model may "simply be able to repeat the correct answer that it previously encountered." To test this they commissioned GSM1k, "a newly constructed collection of 1205 grade school level math problems designed to mirror that of GSM8k," written entirely by human annotators with no language-model assistance.
Construction and matching
Annotators were shown three GSM8K examples and asked to write new problems of similar difficulty, solvable with basic arithmetic and with positive-integer answers. Problems passed three review layers: review by trusted annotators, an independent re-solve without sight of the intended solution (any disagreement discarded the problem), and a general quality audit. Of 2,108 initial problems, 1,419 survived the second solve and 1,375 the audit; the final set of 1,205 was selected after discarding candidates so that the distribution of answer magnitudes matched GSM8K's, since prior work found models struggle with larger numbers. Difficulty was matched by requesting problems with a given number of resolution steps, estimated from calculator tags in GSM8K solutions.
Three checks tested the match. In a human-distinguishability audit, 19 annotators shown four GSM8K problems and one GSM1k problem identified the GSM1k item 21.83% of the time across 1,205 attempts, against 20% by chance. In a timed solve test, 14 annotators solved an average of 4.07 (±0.93) GSM8K problems in 15 minutes and 4.36 (±1.11) GSM1k problems, suggesting GSM1k is "comparable in difficulty (and perhaps even slightly easier)." Two models trained only on pre-2021 data, GPT-NeoX-20B and GPT-2, showed minimal difference between the two sets.
The dataset was not released at publication, with a commitment to release it under the MIT licence when three open-source models of different lineages reach 95% on GSM1k or by June 2025, whichever comes first. A small set of held-out problems that passed quality checks was reserved against leakage through the APIs used to evaluate proprietary models.
Evaluation
Models were run through a fork of EleutherAI's LM Evaluation Harness with a 5-shot prompt drawn from the GSM8K training set, temperature 0 for open models, and the generation limit raised from 256 to 1,000 tokens. API queries ran between 16 April and 10 July 2024. The harness extracts the last number in a response, which the authors note can mark correctly reasoned answers wrong; an ablation with human answer extraction on a subset of high-performing models changed absolute scores but not the pattern of overfitting. Appendix J gives, for example, Claude 3 Opus at 80.2% on GSM8K and 82.4% on GSM1k under automatic extraction and 95.5% and 95.2% under human extraction; GPT-4o at 93.1% and 92.9% (automatic); GPT-4 at 91.1% and 92.3%; Llama 3 70B Instruct at 91.4% and 90.0%; and Llama 3 8B Instruct at 75.2% and 69.0%.
Findings
The paper reports "accuracy drops of up to 8%," and organises its analysis as four lessons. First, "several families of models, including the Phi and Mistral families of models, show systematic tendencies to perform stronger on GSM8k compared to GSM1k for almost every release and scale," with Yi, Xwin, Gemma and CodeLlama showing the pattern to a lesser extent. Second, "all frontier or close-to-frontier models (including the proprietary Mistral Large) appear to perform similarly on both," which the authors attribute more to strong models generalising even if they have seen GSM8K than to unusual care about contamination, noting Mistral Large is the only Mistral model without overfitting. Third, overfit models still reason: Phi-2, despite a 6-point drop, solves over half of GSM1k, comparable to Llama2-70B. Fourth, contamination "is likely not the full explanation": a Spearman rank correlation of 0.36 (p=0.03) links per-character log-likelihood of generating the GSM8K test set to the performance gap, with each percentage point of gap associated with a 1.2×10⁻² increase in log-likelihood, but outliers such as Math-Shepherd-Mistral-7B-RL (highly overfit, low likelihood) suggest other routes, such as reward modelling on synthetic data or selecting models by benchmark performance. Several lesser-known models near the top of the OpenLLMLeaderboard performed "substantially worse on GSM1k," which the authors describe as evidence of Goodhart's law.
Key claims
- GSM1k: 1,205 human-written problems, private at publication, matched to GSM8K on human distinguishability (21.83% vs 20% chance), timed solve rate and answer magnitude.
- Accuracy drops of up to 8 points between GSM8K and GSM1k; Phi and Mistral families systematically overfit; frontier models show minimal overfitting.
- Spearman correlation 0.36 between likelihood of generating GSM8K and the performance gap; contamination is one cause among several.
- All models, including the most overfit, still solve novel grade-school problems.
- The authors recommend that future benchmarks be withheld with a pre-committed release date or condition.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 4 (arxiv.org); the abstract page is arxiv.org.
Relationships
- supports: GSM8K — source of the contamination and overfitting findings and the GSM1k comparison track.
- related: Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021) — the benchmark replicated.
- related: GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024) — Apple's templated approach to the same question, which cites GSM1k as unavailable to researchers.
- related: Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021) — the related-work section summarises functional-evaluation evidence of contamination in MATH.
- related: Scale AI, AI Benchmarks and Evaluation