"GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" is a paper by Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi (Washington State University, during an internship at Apple), Oncel Tuzel, Samy Bengio and Mehrdad Farajtabar of Apple, posted to arXiv as 2410.05229 in October 2024. It introduces a templated variant of GSM8K that regenerates each question with new names and numbers, and uses the resulting distribution of scores to question both the reliability of reported GSM8K results and the reasoning ability they are taken to show. Templates and data are released at github.com/apple/ml-gsm-symbolic.
Summary
The authors state that GSM8K "provides only a single metric on a fixed set of questions," that its popularity raises the risk of contamination, and that its static nature prevents controlled experiments. Citing prior work, they describe the literature as suggesting that "the reasoning process in LLMs is probabilistic pattern-matching rather than formal reasoning," and set out to test this by viewing performance "as a distribution across various problem instances" rather than a point estimate.
Design
GSM8K's test set of 1,319 examples (7,473 training) is the starting point. One hundred test questions were converted into templates in which names, objects and numerical values are variables with domains and conditions (for example divisibility, so that answers stay whole numbers). Automated checks confirm that the original values satisfy the template and reproduce the original answer; 10 random samples per template are reviewed by hand; and after evaluation any question that no two models answer correctly is re-reviewed. Numerical ranges were kept close to GSM8K's so that arithmetic rather than reasoning would not become the bottleneck, which Appendix A.6 checks. Fifty samples per template give 50 datasets of 100 questions each, 5,000 examples in total. Evaluation follows the common GSM8K setup of 8-shot chain-of-thought prompting with greedy decoding; the authors note that the number of shots did not materially change results.
Difficulty variants remove one clause (GSM-Symbolic-M1) or add one or two (P1, P2). GSM-NoOp adds "seemingly relevant but ultimately inconsequential statements," such as noting that five of the kiwis picked were smaller than average.
Twenty-five models were evaluated: more than 20 open models from 2B to 27B parameters (Gemma, Gemma2, Phi, Mistral, Mathstral, Llama3) and the closed GPT-4o-mini, GPT-4o, o1-mini and o1-preview, in nearly 500 evaluations.
Findings
Every model shows non-negligible variance across the 50 sets; the gap between best and worst set exceeds 12 points for Gemma2-9B and is around 15 points for Phi-3.5-mini. For 21 of 25 models the score on the original 100 GSM8K questions lies on the right (high) side of the GSM-Symbolic distribution, "often more than one standard deviation away from the center," which the authors say could be explained by contamination. The drop is larger for models such as Gemma2-9B, Phi-3, Phi-3.5 and Mathstral-7B, and "negligible" for Llama3-8b and GPT-4o. Table 1 in the appendix gives full-GSM8K and GSM-Symbolic accuracies of 95.2% and 94.9% for GPT-4o, 95.1% and 94.5% for o1-mini, 94.9% and 92.7% for o1-preview, 84.9% and 82.1% for Phi-3.5-mini-instruct, and 85.3% and 79.1% for Gemma2-9b-it.
Changing only proper names produces less variance than changing numbers, and the original GSM8K accuracy sits closer to the centre of the names-only distribution. The authors call it "both striking and concerning" that variance exists even for name changes alone. As clauses are added (M1 to P1 to P2), accuracy falls and variance rises consistently across models, with the rate of decline increasing, which they read as consistent with the hypothesis that models are not performing formal reasoning.
On GSM-NoOp there is "a catastrophic performance decline across all tested models," with Phi-3-mini dropping over 65% and o1-preview showing "significant declines"; Table 1 gives NoOp accuracies of 77.4% for o1-preview, 66.0% for o1-mini, 63.1% for GPT-4o and 22.4% for Phi-3.5-mini-instruct. Models "tend to blindly subtract the number of smaller fruits" or read "discount" as multiplication regardless of context. Supplying 8 shots of the same question from GSM-Symbolic (NoOp-Symb), or 8 NoOp examples of other questions (NoOp-NoOp), did not recover performance for most models, which the authors take to indicate "deeper issues in their reasoning processes that cannot be easily mitigated through few-shot learning or fine-tuning."
The conclusion states that the results "indicate that their reasoning is fragile and may be more akin to sophisticated pattern matching rather than true logical reasoning," and that further research is needed toward systems capable of formal reasoning. The related-work section lists iGSM, GSM-IC, GSM-Plus and GSM1K as related datasets, noting that GSM1K "is not publicly available for researchers."
Key claims
- 100 GSM8K test questions templated; 50 sets of 100 generated (5,000 items); 25 models; 8-shot chain-of-thought, greedy decoding.
- All models vary across instantiations; original GSM8K items score on the high side of the distribution for 21 of 25 models.
- Names-only changes perturb less than number changes; adding clauses lowers accuracy and raises variance at an increasing rate.
- GSM-NoOp: drops of up to 65% (Phi-3-mini); o1-preview 77.4%, GPT-4o 63.1%; in-context examples do not repair the drop.
- Stated hypothesis: current LLMs "are not capable of genuine logical reasoning" and replicate reasoning steps seen in training data.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); the abstract page is arxiv.org.
Relationships
- supports: GSM8K — source of the variance, contamination and GSM-NoOp findings on the benchmark.
- contradicts: Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021) — argues that scores on the original benchmark overstate reasoning ability.
- related: A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024) — the contemporaneous Scale AI contamination study, cited as related but private.
- related: Apple
- related: AI Benchmarks and Evaluation