"Measuring Mathematical Problem Solving With the MATH Dataset" is a paper by Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song and Jacob Steinhardt, all of UC Berkeley except Basart (UChicago), posted to arXiv as 2103.03874 in March 2021. It introduces the MATH benchmark of competition mathematics problems and the AMPS pretraining corpus, and reports that accuracy on MATH rises only slowly with model size.
Summary
The paper presents mathematics as "a valuable testbed for problem-solving ability: the ability to analyze a problem, pick out good heuristics from a large set of possibilities, and chain them together to produce an answer," in contrast to "plug-and-chug calculations," which the authors say models can already perform. It argues that "while scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH," and that "simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue."
The MATH dataset
MATH, expanded as the Mathematics Aptitude Test of Heuristics, has 12,500 problems (7,500 training and 5,000 test) drawn from competitions including the AMC 10, AMC 12 and AIME, many of them collected from the Art of Problem Solving contest archive. Problems are tagged with one of seven subjects (Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, Precalculus) and a difficulty level from 1 to 5 "following AoPS," where the first few problems of an AMC 8 exam are often level 1 and AIME problems are level 5. Each problem has a step-by-step solution and a final boxed answer.
Problems and solutions are formatted in LaTeX, with figures encoded in the Asymptote vector-graphics language rather than as images, which the authors say makes it possible "to assess these models on subjects such as geometry for the first time." Final answers follow consistent formatting rules (simplified fractions, alphabetical variable order, decreasing polynomial degree, and so on), and equivalent encodings such as \frac{x}{y} and x/y, or 0.5 and 1/2, are treated as the same, so that the content of the \boxed{} command can be compared with the reference by exact match. Accuracy is the primary metric.
Human-level performance
The paper's human comparison is described as "rough but informative." Twenty problems were sampled from the test set and given to participants with one hour and no calculator. One participant "who does not like mathematics" scored 8/20 (40%); a participant ambivalent toward mathematics 13/20; two participants who like mathematics 14/20 and 15/20; a participant with a perfect AMC 10 score who had attended USAMO several times 18/20; and a three-time IMO gold medalist 18/20 (90%), with the missed questions "exclusively due to small errors of arithmetic." The introduction describes the 40% participant as "a computer science PhD student who does not especially like mathematics." Appendix A.6 states there were six participants, all university students studying computer science, four of them authors of the paper, and that the study "was only to give a rough sense of human-level performance." The authors add that "expert-level performance is theoretically 100% given enough time."
AMPS
The Auxiliary Mathematics Problems and Solutions corpus contains over 100,000 Khan Academy problems with step-by-step solutions across 693 exercise types, and approximately 5 million problems generated from 100 hand-designed Mathematica scripts (37 with full step-by-step solutions), around 50,000 per script, for over 23 GB of problems and solutions.
Experiments
Models are GPT-2 at 0.1B to 1.5B parameters, pretrained on AMPS and fine-tuned on MATH, and GPT-3 Curie (about 13B) and Davinci (about 175B) via the OpenAI API, fine-tuned or few-shot with 8 example problems. By default models are prompted to output the final answer directly. Table 2 reports average accuracy of 5.4% for GPT-2 0.1B, 6.2% for 0.3B, 6.4% for 0.7B, 6.9% for 1.5B, 3.0% for few-shot GPT-3 13B, 5.6% for fine-tuned GPT-3 13B, and 5.2% for few-shot GPT-3 175B; the introduction summarises the range as 3.0% to 6.9%. A 15x increase in GPT-2 parameters gives a 28% relative improvement, which the authors read as accuracy "only increasing slowly, quite unlike most other text-based tasks." GPT-2 1.5B reaches around 15% on level 1 problems and around 4% on level 5.
AMPS pretraining lets a 0.1B GPT-2 (5.4%) match a fine-tuned 13B GPT-3 without AMPS (5.2%), which the authors equate to "about as much as a 130x increase in parameters." Adding Math StackExchange data to AMPS pretraining lowered accuracy from 6.2% to 6.0%. Having GPT-2 1.5B generate a full step-by-step solution before the answer reduced accuracy to 5.3%, which the authors attribute to "a snowballing effect, in which partially generated 'solutions' with mistakes can derail subsequent generated text"; training on solutions, by contrast, helped (removing them lowered 1.5B accuracy by 0.6 points to 6.3%), and providing 99% of the reference solution as a hint raised GPT-2 0.7B to approximately 40%. Confidence-based error detection with GPT-2 1.5B gave an AUROC of 68.8%. BART-Large pretrained on AMPS scored 4.9%.
The conclusion states that "if trends continue, the community will need to discover conceptual and algorithmic breakthroughs to attain strong performance on MATH."
Key claims
- MATH: 12,500 competition problems, 7,500 train and 5,000 test, seven subjects, five AoPS-style difficulty levels, LaTeX and Asymptote formatting, boxed final answers scored by exact match after normalisation.
- Model accuracy at publication: 3.0% to 6.9%; up to about 15% on the easiest level.
- Informal human study on 20 problems in one hour: 40% for a computer-science PhD student who does not especially like mathematics, 90% for a three-time IMO gold medalist; six participants, four of them authors.
- AMPS pretraining is worth roughly a 130x parameter increase at the scales tested.
- Generating step-by-step solutions at test time decreased accuracy for the models tested; training on solutions increased it.
- Stated position: scaling alone is not solving MATH, and algorithmic advances will be needed.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); the abstract page is arxiv.org.
Relationships
- supports: MATH and MATH-500 — the primary text behind the benchmark's description, construction, original results and human study.
- related: Training Verifiers to Solve Math Word Problems (Cobbe et al., OpenAI, 2021) — the contemporaneous easier dataset that positions itself against MATH.
- related: Let's Verify Step by Step (Lightman et al., OpenAI, 2023) — the OpenAI paper that introduced the MATH-500 subset and reported 78% on it.
- related: A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024) — later contamination analysis covering MATH as well as GSM8K.
- related: Dan Hendrycks — lead author.
- instance-of: AI Benchmarks and Evaluation