GSM8K

GSM8K (Grade School Math 8K) is a dataset of about 8,500 arithmetic word problems written by human contractors, released by OpenAI in October 2021 alongside the paper "Training Verifiers to Solve Math Word Problems" Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021). Each problem takes between two and eight steps of elementary arithmetic to solve, and a model is scored on whether its final number matches the reference answer. In the original paper a fine-tuned GPT-3 solved roughly a third of the test problems, and the paper's verifier method about 55%; by mid-2024 developers were reporting scores above 95%, and OpenAI wrote in September 2024 that frontier models "do so well on MATH and GSM8K that these benchmarks are no longer effective at differentiating models" (Source: openai.com). The benchmark remains in use for smaller models and as the base for several derivative tests of contamination and robustness.

What it measures

The authors present GSM8K as a probe of multi-step mathematical reasoning in language models, at a level where the mathematics itself is easy and the difficulty lies in reading the problem correctly and chaining several calculations without error. The paper states that "a bright middle school student should be able to solve every problem," that the problems "require no concepts beyond the level of early Algebra," and that GSM8K's "main difficulty lies in both properly interpreting a question and reasoning through the steps to solve it" Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021). OpenAI's accompanying post described grade-school mathematics as a testbed for a broader weakness: a model can "mimic the cadence of correct solutions" while producing errors in logic, and "one subtle mistake is enough to derail an entire solution" (Source: openai.com).

The dataset was built to support a specific method. The paper trains verifiers, separate models that score candidate solutions, and at test time samples 100 solutions per problem and returns the one the verifier ranks highest. The authors report that a 6-billion-parameter model with a verifier slightly outperformed a fine-tuned 175-billion-parameter model, a gain they describe as roughly equivalent to a 30-fold increase in model size Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021).

Design

Dataset construction and provenance

The problems were written by people rather than scraped. OpenAI first collected about a thousand problems from freelance contractors hired through Upwork, then scaled collection through the data-labeling company Surge AI. Contractors were given seed questions generated by a few-shot-prompted 175B GPT-3 model, which they could use directly, adapt, or ignore, and were instructed not to reuse problem settings or templates; OpenAI computed pairwise similarity scores between problems and fed them back to the writers Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021).

For quality control, workers re-solved every problem without seeing the original solution, and problems producing disagreement were repaired or discarded. A second agreement check on a smaller subset found that 1.7% of problems still produced disagreements, which the authors give as their estimate of the fraction containing "breaking errors or ambiguities," adding that "it is possible that a larger percentage of problems contain subtle errors" Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021).

Task format

Each item is a short word problem with a natural-language solution ending in a single number. An example from OpenAI's release post: "Tim grows 5 trees. Each year he collects 6 lemons from each tree. How many lemons does he get in a decade?" (Source: openai.com). Reference solutions carry calculator annotations of the form <<20+10=30>>, which the original models were trained to emit so that a calculator could override their arithmetic; the annotations were generated automatically rather than by the contractors. The release also includes a "Socratic" version of the data in which each solution step is preceded by an automatically generated sub-question (Source: github.com).

Splits, scoring and metric

The dataset is divided into 7,473 training problems and 1,319 test problems GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024); the paper rounds these to 7.5K and 1K. Both splits, including the test answers, are public in OpenAI's GitHub repository, and no private hold-out exists (Source: github.com).

A response is correct if its final answer equals the reference number. The paper uses "test@N" for the share of problems solved at least once in N attempts. Later reporting varies the prompt rather than the grading: the most common settings are few-shot chain-of-thought prompting (several worked examples placed in the prompt, with the model asked to reason step by step), zero-shot chain-of-thought, and maj1@k or self-consistency (sample k solutions and take the most common final answer). These settings produce different numbers for the same model and are recorded in the Results tables.

Human baseline

OpenAI reported that "a small sample of 9-12 year olds scored 60% on a test from our dataset," against 55% for its verifier system on the same problems (Source: openai.com). The paper itself reports no human accuracy figure. Scale AI's 2024 replication had 14 annotators solve as many GSM8K problems as they could in 15 minutes; they averaged 4.07 (standard deviation 0.93), a speed measure rather than an accuracy ceiling A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024).

Versions and variants

• GSM1k (Scale AI, May 2024): 1,205 newly written problems matched to GSM8K on step count, answer magnitude and human solve rate, written without any language-model assistance and kept private to measure overfitting. Annotators asked to pick the GSM1k problem out of five could do so 21.83% of the time against a 20% chance rate. Scale committed to releasing the set once three open-source model lineages reached 95% on it or by June 2025, whichever came first A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024).
• GSM-Symbolic (Apple, October 2024): 100 GSM8K test questions converted into templates whose names and numbers can be resampled, yielding 50 generated sets of 100 questions each, plus harder variants with clauses removed or added (M1, P1, P2) and GSM-NoOp, which adds a clause that appears relevant but does not affect the answer GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024).
• GSM8K-Platinum (MIT, March 2025): a revision of the full test set in which 219 questions flagged by model disagreement were inspected by hand; 110 were removed, 99 verified, and 10 had their answers corrected (Source: gradientscience.org).
• MGSM: a multilingual translation of GSM8K problems, reported in Anthropic's Claude 3 model card among others (The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024)).
• The GSM-Symbolic authors also list iGSM, GSM-IC and GSM-Plus as related variants by other groups GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024).

Development and maintainers

The paper's twelve authors were all at OpenAI: Karl Cobbe and Vineet Kosaraju (equal contribution), Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse and John Schulman; the acknowledgements thank Dan Hendrycks among others for feedback and "the team at Surge AI for performing the GSM8K data collection" Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021). The dataset has not been revised since release. There is no official leaderboard and no verification of submitted scores: results are self-reported by developers in papers and model cards, each under its own prompt. Epoch AI's Benchmarking Hub lists GSM8K as "developer administered" and collects reported scores for 235 models (Source: epoch.ai).

Results

The original paper reports its results only as charts. Researchers at MIT who re-examined the benchmark read the best fine-tuned baseline off the paper's Figure 2 as approximately 34% (Source: arxiv.org), and Google researchers cite 55% as the prior best, for OpenAI's fine-tuned GPT-3 175B with a verifier (Source: arxiv.org). Chain-of-thought prompting took PaLM 540B from 17.9% to 56.9% in January 2022 without any fine-tuning, majority voting over sampled solutions reached 78.5% with Minerva in June 2022, and GPT-4 reported 92.0% in March 2023 (Source: arxiv.org; Source: arxiv.org; GPT-4 Technical Report (OpenAI, arXiv 2303.08774, March 2023)). Developer-reported scores reached 95.0% with Claude 3 Opus in March 2024 (The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024)) and 96.8% with Llama 3.1 405B that July. OpenAI's o1 announcement in September 2024 did not report GSM8K, and the MIT group noted in early 2025 that recent frontier releases "including o1 and Claude 3.7 Sonnet" had excluded it (Source: gradientscience.org).

Original test set

Accuracy on the 1,319-question public test set. Prompting and sampling differ by row.

GSM1k

Scale AI's private 1,205-problem replica, run under the same standardized 5-shot harness as its GSM8K measurements above. A score close to the same model's GSM8K score indicates little overfitting to the public test set.

GSM-Symbolic

Mean accuracy across 50 generated sets of 100 questions built from GSM8K templates with resampled names and numbers.

GSM-NoOp

GSM-Symbolic questions with one added clause that looks relevant but does not change the answer.

GSM8K — 8-shot, CoT

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8K — Pre-trained

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8K — Pre-trained, 16-shot long context

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8k (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8k — maj1@k (k=100)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8k — maj1@k (k=64)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

GSM8K-Platinum (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Answers are free-form numbers, so guessing contributes almost nothing and the floor is effectively zero; the smallest models tested in the chain-of-thought paper score below 5% (Source: arxiv.org).

The ceiling is below 100%. The authors' own estimate is that 1.7% of problems contain breaking errors or ambiguities. The MIT revision removed 110 of the 1,319 test questions as ambiguous or internally inconsistent and corrected the answers to 10 more, and its authors argue that the apparent plateau of frontier models at about 95% is "in large part caused by label noise." Their illustration is that Claude 3.7 Sonnet with extended thinking and Llama 3.1 405B each made 45 errors on the original test set, while on the revised set the former made 2 and the latter 17 (Source: gradientscience.org). One of the flawed items they cite gives a number of stalls as both ten and twenty within the same question (Source: arxiv.org).

Scores are sensitive to how the answer is read out of the response. In Scale AI's standardized harness, which takes the last number in the output, Claude 3 Opus scored 80.2%; when human annotators extracted the answers from the same kind of output it scored 95.5%. Scale notes that its GSM8K numbers "may occasionally be below the reported model benchmark numbers" because it used one prompt for all models rather than each developer's preferred prompt A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024). Sampling matters as well: Minerva 540B moves from 58.8% to 78.5% when the most common answer of 40 samples is taken instead of a single sample (Source: arxiv.org).

A developer-reported score can also reflect training on GSM8K's own training split. OpenAI disclosed that part of that split was mixed into GPT-4's pre-training data and recommended reading GPT-4's 92.0% "as something in-between true few-shot transfer and full benchmark-specific tuning" (GPT-4 Technical Report (OpenAI, arXiv 2303.08774, March 2023)).

Limitations and criticism

Contamination and overfitting

Scale AI found accuracy drops of up to 8 percentage points between GSM8K and its private replica, concentrated in particular model families: Phi and Mistral models "show systematic tendencies to perform stronger on GSM8k compared to GSM1k for almost every release and scale," with Yi, Xwin, Gemma and CodeLlama showing the pattern to a lesser extent. The gap correlated with a model's likelihood of reproducing GSM8K test examples (Spearman 0.36, p = 0.03), which the authors read as evidence of partial memorization. They also report that "all frontier or close-to-frontier models" showed minimal overfitting, that the most overfit models still solved many unseen problems (Phi-2 solved over half of GSM1k despite a 6% drop), and that contamination was "likely not the full explanation," pointing to indirect routes such as training on similar data or selecting checkpoints by benchmark score A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024).

Fragility

Apple's researchers report that accuracy varies across regenerated versions of the same question, that the score on the original GSM8K items sat on the high side of the regenerated distribution for 21 of 25 models, and that adding one irrelevant clause reduced accuracy by up to 65% for some models. They hypothesize that "current LLMs are not capable of genuine logical reasoning" and instead "replicate the reasoning steps observed in their training data" GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024). In their own table the drop on regenerated questions is small for the strongest models tested (GPT-4o 95.2% on GSM8K against 94.9%) and larger for small open models, while the irrelevant-clause test lowered every model's score.

Saturation

OpenAI stated in September 2024 that frontier models do so well on GSM8K and MATH "that these benchmarks are no longer effective at differentiating models" and moved its headline mathematics reporting to AIME (Source: openai.com). The FrontierMath authors and the MathArena authors both describe GSM8K as near or at saturation in motivating their own benchmarks (Source: FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., Epoch AI, 2024); Source: MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025)). The MIT group's response is that a cleaned version still separates models on reliability, since every model it tested made at least one genuine error on elementary problems (Source: gradientscience.org).

Construct

The Apple paper notes that GSM8K "provides only a single metric on a fixed set of questions," which limits what can be learned about where reasoning fails GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (Mirzadeh et al., Apple, 2024).

Use in policy and safety frameworks

GSM8K appears in developer model cards and technical reports as a general capability indicator rather than in any safety threshold: the GPT-4 technical report, Anthropic's Claude 3 model card and Meta's Llama 3 paper all report it (GPT-4 Technical Report (OpenAI, arXiv 2303.08774, March 2023); The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024); Source: arxiv.org). Anthropic's launch announcement for Claude 3 cited "basic mathematics (GSM8K)" among the benchmarks on which Opus led its peers (Source: anthropic.com). The LLaMA paper of February 2023 used it to compare an open model with Google's Minerva (LLaMA: Open and Efficient Foundation Language Models (Touvron et al., Meta AI, February 2023)). No use of GSM8K in a responsible scaling policy, preparedness framework, regulation or standard was found.

Relationships

• instance-of: AI Benchmarks and Evaluation
• related: MATH and MATH-500 — the harder competition-mathematics dataset released the same year, which the GSM8K paper cites as too difficult to measure progress on at the time
• related: AIME (as a model benchmark) — the benchmark OpenAI moved to for headline mathematics reporting in September 2024
• related: FrontierMath, MathArena — later benchmarks whose authors cite GSM8K's saturation and contamination as motivation
• related: OpenAI — creator
• related: Scale AI, Apple — authors of the GSM1k and GSM-Symbolic studies
• related: Claude 3 Opus, Llama 3 (Meta), GPT-4 Family (OpenAI) — models whose launch reporting included GSM8K

See also

• MMLU
• HumanEval
• AI Progress Measurement