MATH and MATH-500

MATH (the name abbreviates "Mathematics Aptitude Test of Heuristics") is a dataset of 12,500 problems drawn from high-school mathematics competitions, introduced in March 2021 by Dan Hendrycks and co-authors at UC Berkeley and the University of Chicago Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021). A model must produce the exact final answer to each problem. The strongest model in the original paper scored 6.9%; OpenAI's o1 reported 94.8% in September 2024 on MATH-500, a 500-problem subset of the test set that OpenAI had carved out in 2023, and OpenAI wrote at that point that the benchmark was "no longer effective at differentiating models" (Source: openai.com). The full dataset was removed from Hugging Face in January 2025 following a copyright takedown notice from Art of Problem Solving.

What it measures

The authors describe MATH as a measure of mathematical problem-solving ability, distinguished from earlier datasets of "plug-and-chug" arithmetic. Most of its problems, they write, "cannot be solved with a straightforward application of standard K-12 mathematics tools"; humans solve them by applying problem-solving techniques and heuristics Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021). The paper's argument at release was about scaling: accuracy rose only from 5.4% to 6.9% across a fifteen-fold increase in model size, from which the authors concluded that "simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue" and that algorithmic advances would be needed Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021).

Design

Dataset construction and provenance

The paper states that the dataset "consists of problems from mathematics competitions including the AMC 10, AMC 12, AIME, and more," that "many of these problems can be collected from aops.com/community/c3158_usa_contests," and that the competitions "span decades." There are 7,500 training problems and 5,000 test problems, each with a full step-by-step solution written in LaTeX, with geometry diagrams encoded in the Asymptote vector-graphics language so that the whole task is text Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021).

Problems are tagged with one of seven subjects (Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, Precalculus) and a difficulty level from 1 to 5, "following AoPS": the first few problems of an AMC 8 exam are often level 1, while AIME problems are level 5 Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021). Alongside MATH the authors released AMPS, a 23 GB pretraining corpus of over 100,000 Khan Academy problems and about 5 million Mathematica-generated problems.

Task format

An example of an easier item from the paper: "Tom has a red marble, a green marble, a blue marble, and three identical yellow marbles. How many different groups of two marbles can Tom choose?" The reference solution works through two cases and ends with the answer, 7, inside a LaTeX \boxed{} command Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021).

Scoring and metric

The authors designed the answers to be "unique after normalization, allowing MATH to be scored with exact match rather than with heuristic metrics" Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021). In practice the grader must decide whether two expressions such as "5/2" and "2.5" are the same, and implementations differ. Epoch AI's Benchmarking Hub runs three scorers side by side (normalized string match, symbolic equivalence via the SymPy library, and a model-based judge) and plots the model-judged figure; a response that does not follow the required answer format receives no credit (Source: epoch.ai). Developer reports vary the prompt (zero-shot or few-shot chain of thought) and the sampling: pass@1 is accuracy from a single sample per problem, and maj@k takes the most common answer of k samples.

Human baseline

The paper's human evaluation is informal. Six university computer-science students, four of them authors of the paper, each had one hour to answer 20 sampled problems without a calculator. A participant described as "a computer science PhD student who does not especially like mathematics" scored about 40%, and a three-time International Mathematical Olympiad gold medalist scored 90% Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021).

Splits and contamination controls

All problems and solutions, including the test set, were published openly. The source competitions are also discussed online independently of the dataset. OpenAI's 2023 paper acknowledged that the test set "contains problems that are discussed in several online venues, and it is likely that some of these problems appear in the pretraining dataset for our models," and that string-matching decontamination could not give strong guarantees Let's Verify Step by Step (Lightman et al., OpenAI, 2023).

Versions and variants

• MATH-500. For the paper "Let's Verify Step by Step" (May 2023), OpenAI moved 4,500 of the 5,000 MATH test problems into its training data to avoid overfitting on the 7,500 training problems, and evaluated only on the remaining 500, selected uniformly at random Let's Verify Step by Step (Lightman et al., OpenAI, 2023). OpenAI's o1 announcement states that its MATH evaluation "used the same 500 problem test split," and later developer reports follow that convention (Source: openai.com). The subset is distributed on Hugging Face as HuggingFaceH4/MATH-500 (Source: huggingface.co).
• MATH Level 5. Epoch AI evaluates models on the 1,324 level-5 questions in the test set, chosen "because it's not yet saturated by current models" at the time the series began and to avoid running all 5,000 questions (Source: epoch.ai).
• MATH-Perturb (Huang et al., February 2025): 279 level-5 problems rewritten two ways, with simple perturbations that preserve the solution method (MATH-P-Simple) and hard perturbations that invalidate it (MATH-P-Hard) (Source: arxiv.org).
• A functional variant of MATH, in which each problem is a program that generates fresh instances, was proposed by Srivastava et al.; Scale AI's GSM1k paper summarizes its results as suggesting "substantial amounts of data contamination, including for frontier models, in the MATH dataset" A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al., Scale AI, 2024).

Development and maintainers

The authors are Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song and Jacob Steinhardt; the paper appeared in the NeurIPS 2021 Datasets and Benchmarks track Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021). Hendrycks also co-designed MMLU and later co-founded the Center for AI Safety. There is no official leaderboard: results are self-reported by developers, and independent series are run by Epoch AI (Level 5) and were run by the evaluation company Vals AI (MATH-500) until it archived the benchmark.

Copyright takedown

Art of Problem Solving (AoPS), the education company whose forum the paper names as a source, sent Hugging Face a takedown notice dated January 2, 2025. The notice states that AoPS compared the dataset with its Alcumus learning platform and "found that over 10,000 of the 12,500 problems have a similarity score of 95 or above," that some problems come from competitions such as MATHCOUNTS and the AMC "with permission to use in our system while others are written directly by our curriculum development team," and that all solutions "are entirely proprietary and written by our curriculum development team" (Source: huggingface.co). The dataset's Hugging Face page now reads "Access to this dataset has been disabled" with a reference to the DMCA notice (Source: huggingface.co), and an issue opened on January 29, 2025 in the UK government's Inspect Evals repository recorded the removal (Source: github.com). No public response from the dataset's authors was found.

Results

The 2021 paper reported accuracies between 3.0% and 6.9% for the language models it tested. Google's Minerva reached 50.3% in June 2022 with majority voting over 64 samples, GPT-4 was credited with 42.5% zero-shot, and Claude 3 Opus reported 60.1% in March 2024 (Source: arxiv.org; The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024)). On the 500-problem subset, OpenAI's process-supervised reward model selected correct solutions for 78.2% of problems in May 2023, o1 reported 94.8% in September 2024, and DeepSeek-R1 reported 97.3% in January 2025 (DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning). By mid-2025 developer tables placed most frontier models between 96% and 99%. Vals AI's independently run leaderboard, which topped out at 96.40%, was archived with the note that "performance on this benchmark has saturated" (Source: vals.ai). On Epoch's harder Level 5 subset the highest listed score is 98.1%.

MATH-500

The 500-problem subset of the test set defined in OpenAI's 2023 paper.

Full test set

All 5,000 test problems.

Level 5 (Epoch AI)

The 1,324 hardest test problems, run by Epoch AI with a fixed prompt, pass@1, scored by a model-based equivalence judge. The hub does not show a run date for each entry; rows carry the date the leaderboard was read.

MATH (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MATH — 0-shot, CoT

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MATH — Pre-trained

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MATH — maj1@k (k=256)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Math Instruct (0-shot, no CoT) (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Math Instruct (0-shot, CoT) (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MATH-CoT (as reported in developer documents)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Answers are free-form expressions, so the chance level is effectively zero. No published estimate of the label-error rate in MATH was found, which leaves the practical ceiling unknown; independent leaderboards have topped out at 96–98%.

In human terms the only reference points are the two participants in the authors' informal study, about 40% and 90% on a 20-problem sample. The authors call the exercise "only ... a rough sense of human-level performance" Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021).

Numbers from different tracks are not interchangeable. OpenAI's o1 post gives GPT-4o 60.3% on MATH-500, while the DeepSeek-R1 paper gives a GPT-4o snapshot 74.6% on the same subset and Meta's Llama 3 paper gives GPT-4o 76.6% on the full test set; the differences reflect prompts, model snapshots and graders rather than the problems. For mid-2025 models, developer-reported MATH-500 figures cluster at 98–99% while Vals AI's runs of the same models sit at 94–96%. Sampling changes the figure substantially at lower capability levels: Minerva 540B moves from 33.6% with one sample to 50.3% with majority voting over 64, and OpenAI's 2023 result of 78.2% depended on sampling 1,860 solutions per problem and selecting one with a trained reward model.

Limitations and criticism

Contamination

The test problems and their worked solutions have been public since 2021 and the underlying competition problems for longer. OpenAI's 2023 paper said some degree of contamination "may have slightly inflated our performance on the MATH test set" while arguing that relative comparisons within the paper were unaffected Let's Verify Step by Step (Lightman et al., OpenAI, 2023). Vals AI states that the MATH-500 questions are "all in the public domain, creating a high risk of pre-training on the test set," and that results may "no longer reflect a meaningful difference in model capabilities" (Source: vals.ai). The practice of moving 4,500 test problems into training data, which created MATH-500, means those problems are training material for any developer using the PRM800K data.

Memorized methods

The MATH-Perturb authors report "significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%)" when level-5 problems were altered so that the original solution method no longer applied, and describe "a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts" (Source: arxiv.org).

Saturation

OpenAI's September 2024 statement that frontier models "do so well on MATH and GSM8K that these benchmarks are no longer effective at differentiating models" accompanied its move to AIME for headline reporting (Source: openai.com). The FrontierMath authors cited near-perfect performance on MATH and GSM8K as a reason to build a research-level benchmark, and the MathArena authors describe both as "largely ... saturated" (Source: FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., Epoch AI, 2024); Source: MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025)).

Provenance and licensing

The AoPS takedown notice characterizes the dataset as "entirely pirated" from its Alcumus platform, including solutions it describes as proprietary. That is the rights-holder's allegation in a DMCA notice, not a court finding; its practical effect was the removal of the canonical copy from Hugging Face and documentation changes in downstream evaluation suites (Source: huggingface.co; Source: github.com).

Format

Exact-match grading of final answers gives no credit for reasoning and no penalty for a correct answer reached by flawed steps, a limitation the MathArena authors raise for final-answer benchmarks generally MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Use in policy and safety frameworks

MATH has served as a general capability indicator in developer documentation, including the GPT-4 technical report (which discloses that MATH training data was mixed into GPT-4's pre-training), Anthropic's Claude 3 model card and Meta's Llama 3 paper (GPT-4 Technical Report (OpenAI, arXiv 2303.08774, March 2023); The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024); Source: arxiv.org). Epoch AI uses the Level 5 subset as one of the benchmark series in its Benchmarking Hub (Source: epoch.ai). Distillation research has used MATH-500 as an outcome measure, for example the report that reasoning-trace distillation raised a fine-tuned Qwen2.5-7B-Instruct from 68.4% to 76.0% (Stealing Reasoning Traces from Proprietary LLM APIs). No use of MATH in a capability threshold, regulation or standard was found. The takedown is an instance of a copyright claim against an evaluation dataset rather than against model training.

Relationships

• instance-of: AI Benchmarks and Evaluation
• related: GSM8K — the easier companion benchmark of the same period
• related: AIME (as a model benchmark) — MATH's level-5 problems include past AIME problems; AIME replaced MATH in headline reporting from late 2024
• related: FrontierMath, MathArena — successors motivated in part by MATH's saturation and contamination
• related: MMLU — by the same lead author
• related: Dan Hendrycks — lead author
• related: Epoch AI — runs the Level 5 series
• related: OpenAI o-series (o1 → o4-mini), DeepSeek-R1 — reasoning models whose launch reporting used MATH-500
• related: Distillation — MATH-500 as an outcome measure in distillation studies

See also

• Humanity's Last Exam
• AI Progress Measurement