"FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI" is the paper introducing the FrontierMath benchmark, posted to arXiv as 2411.04872 in November 2024 and revised through version 7 (December 2025). Its core contributors are Elliot Glazer, Ege Erdil and Tamay Besiroglu of Epoch AI, with Olli Järviniemi, and its author list runs to about 40 names, most of them contributing mathematicians at universities including King's College London, MIT, UC Berkeley, Harvard, Cornell and ETH Zurich. Epoch staff Jaime Sevilla, Anson Ho, Matthew Barnett and others are also co-authors. The acknowledgements thank Terence Tao for contributing several problems and state that the authors "gratefully acknowledge OpenAI for their support in creating the benchmark."
Summary
The paper motivates the benchmark with two limitations of existing mathematics evaluations. Standard benchmarks such as MATH and GSM8K "primarily assess competency at the high-school and early undergraduate level" and are near-perfectly solved, so there is no rigorous way to evaluate capabilities in domains requiring "deeper theoretical understanding, creative insight, and specialized expertise." And public benchmarks suffer from data contamination, while fresh competition problems such as the IMO or AIME are few and costly to grade. FrontierMath answers both with problems that are new and unpublished, automatically verified, and span "most branches of mathematics" in the MSC2020 classification (70% of top-level subjects, excluding general, historical and educational codes).
Construction
The problems were written by over 60 mathematicians "from universities across more than a dozen countries," ranging from graduate students to faculty, who collectively hold 14 IMO gold medals; one contributor holds a Fields Medal. Four requirements governed problem creation: originality (problems could build on existing ideas only "in novel and non-obvious ways"); automated verifiability (answers are integers or objects expressible in SymPy, checked by script); guessproofness (as a rule of thumb, no more than a 1% chance of guessing the answer "without doing most of the work"); and computational tractability (any scripts needed to reach the answer must run in under a minute on standard hardware, starting from standard knowledge of the field). The authors state that the verifiable-answer format means the benchmark "cannot include problems that require mathematical proofs or formal reasoning steps," and explain why they did not accept Lean proofs: models are not generally trained to write them, and mathlib does not fully cover the undergraduate curriculum.
Every problem received blind peer review by at least one mathematician checking correctness, ambiguity, guessproofness and metadata. To estimate residual error, second reviews were commissioned on 30 randomly chosen problems, and 5 further problems were removed to serve as public samples, so 35 accepted problems received scrutiny beyond the first review. Two of the 35 had an incorrect author-provided answer undetected in first review; a Jeffreys-prior posterior gives about 6.9%, and allowing for errors the second review may also have missed, the authors conclude that "estimating a critical error rate of approximately 10% is reasonable," comparing this with a reported label-error rate above 6% in ImageNet and above 9% in MMLU. Six of the 35 had missing hypotheses and two were judged guessable with less effort than intended; all were corrected. The authors describe moving from a "passive" to an "active" review process requiring concrete confirmation of soundness.
Originality was checked by expert review and by the plagiarism tools Quetext and Copyscape, chosen "because they posed the lowest risk of our problems being stored in databases that might later be used to train AI models." Submissions were encouraged through encrypted channels, though the paper acknowledges that "standard email clients were used when communicating about a subset of the problems outside our immediate team."
Each problem carries three author-rated difficulty dimensions, reviewed by peers: background (1, high school, to 5, research level), creativity (hours for an expert to find the key ideas) and execution (hours to compute the answer once found). The authors call these "rough guidance," noting that first and second reviewers' ratings "rarely matched."
Composition
Number theory (17.8% of MSC tags) and combinatorics (15.8%) are the largest subjects, together about 34% of tags; group theory is 8.9%. Counted by problem, number theory appears in 44% of problems, combinatorics in 39% and group theory in 22%. Over 200 distinct techniques are listed across solutions, none in more than 5% of problems.
Evaluation
Models interact with a Python environment, running code and reading its output, until they submit a final answer in a prescribed format or reach a 10,000-token limit, after which a final prompt demands an immediate answer. Six models were evaluated: o1-preview, o1-mini, GPT-4o (2024-08-06), Claude 3.5 Sonnet (2024-10-22), Grok 2 Beta and Gemini 1.5 Pro 002. On a single run "no model achiev[ed] even a 2% success rate," and only four problems were solved by any model. Repeated five-run trials on those four found high variability; only o1-preview on one problem succeeded in all five runs. The paper cautions that with so few successes "the precise ordering of model performance should be interpreted with significant caution," and that a correct answer does not imply correct reasoning, since on one problem "running a few simple simulations was sufficient to make accurate guesses." Models differed in behaviour: o1-preview averaged 1.29 responses per problem and Grok 2 Beta 3.81; Gemini 1.5 Pro 002 hit the token limit on 16.8% of questions while Claude 3.5 Sonnet, GPT-4o and Grok 2 Beta exceeded it in more than 45% of attempts. Version 7 does not state how many problems the evaluated set contained.
Interviews and limitations
Interviews with Terence Tao, Timothy Gowers, Richard Borcherds and Evan Chen (a co-author) found all four characterising the problems as exceptionally challenging; Tao is quoted expecting the benchmark to "resist AIs for several years at least," and Borcherds observing that the problems "aren't quite the same as coming up with original proofs." The discussion lists as limitations the exclusion of proof-writing and open-ended exploration, the gap between hours-long problems and research spanning months, and the fact that current models solve so few problems that the benchmark "temporarily limits" comparisons between them.
Key claims
- Problems are original, unpublished, automatically verified via SymPy, guessproof at roughly the 1% level, and computable in under a minute.
- Over 60 contributing mathematicians, 14 IMO gold medals and one Fields Medal among them; OpenAI supported the benchmark's creation.
- Estimated critical error rate approximately 10% (2 wrong answers in 35 second-reviewed problems).
- Six leading models each scored under 2% on a single run with a 10,000-token budget; four problems solved in total.
- The requirement of numerical answers "differs from typical mathematical research practice," per the interviewed mathematicians.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 7 (arxiv.org); the abstract page is arxiv.org. Sample-problem transcripts are linked from the paper at epochai.org.
Relationships
- supports: FrontierMath — the primary text behind the benchmark's design, review process and initial results.
- related: Is AI already superhuman on FrontierMath? (Anson Ho, Epoch AI, 2025) — Epoch's later human-baseline study on the benchmark.
- related: Announcing FrontierMath Erdős (Epoch AI, September 2026) — the later open-problems extension.
- related: Are We Done with MMLU? (Gema et al., 2024) — cited for the MMLU error rate the authors compare their own against.
- related: Epoch AI, OpenAI
- instance-of: AI Benchmarks and Evaluation