"MathArena: Evaluating LLMs on Uncontaminated Math Competitions" is a paper by Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović and Martin Vechev of ETH Zurich and INSAIT, Sofia University, posted to arXiv as 2505.23281 in May 2025 and revised through version 3 (January 2026). It describes the MathArena platform, which evaluates language models on recurring mathematics competitions immediately after their problems are released, and reports results across seven 2025 competitions. Code and model responses are released at github.com/eth-sri/matharena and results at matharena.ai.
Summary
The paper opens with three concerns about existing mathematics benchmarks. Public competition datasets such as AIME 2024 are "susceptible to data contamination," whether through inclusion in training data or through use for hyperparameter tuning. Private benchmarks such as FrontierMath and Humanity's Last Exam avoid leakage but raise "concerns about reproducibility and transparency," may grant access selectively, and are costly (the authors note HLE's $500,000 prize pool). And most benchmarks score only final answers, which "can be misleading, as models may arrive at the correct answer through pattern recognition or brute-force enumeration, rather than genuine mathematical reasoning." MathArena's answer is that "recurring math competitions provide a stream of high-quality, challenging problems that can be used for real-time evaluation," pre-vetted for originality by competition organisers, so that evaluating models released before a competition "eliminates the risk of contamination."
Pipeline
Table 1 lists the competitions covered: AIME (30 problems, February), HMMT February (30), USAMO (6 proof problems, March), BRUMO (30, April), CMIMC (40, May), IMO (6 proof problems, July) and Project Euler (20 or more recent problems), with MMATHS, DMM, PUMAC and Putnam planned. Final-answer competitions were chosen for difficulty; Kangaroo was tried and excluded as "already saturated." Models are added only if they compete for the top score, the best open-weight score, or a Pareto-optimal cost-performance point; non-reasoning models are excluded. Each model is run with provider-recommended hyperparameters, four responses per problem, and the average reported as pass@1 with no majority voting. Models released after a competition are marked on the leaderboard. Project Euler runs allow Python and C++ execution, capped at 20 executions per problem.
Final answers are extracted from a boxed environment by a custom rule-based LaTeX-to-SymPy parser, cross-checked by a Gemini-2.5-Flash judge for semantic equivalence and by manual review of flagged outputs. Proofs are graded by expert humans against a grading scheme written to resemble the competition's own, with two independent judges per solution. Because competitions are small, the paper reports 95% confidence intervals on accuracy (treating each answer as a Bernoulli trial) and rank intervals from a paired permutation test.
Results
Over 50 models were evaluated across the seven competitions (162 problems) at roughly $2,000 in API costs. On the four final-answer competitions (130 problems), the best non-deprecated models at the time of writing were GPT-5 (high) at 91.3% average accuracy, Grok 4 at 90.6% and Grok 4 Fast at 90.4%; the authors write that these "vastly outperform the top 1% of human participants in AIME and HMMT." Among open-weight models GPT-OSS-120B led at 89.3%, followed by DeepSeek-v3.2 (Think) at 88.3%. Table 3 gives rank intervals of 1–4 for GPT-5 and 1–5 for the two Grok models, with Gemini-2.5-Pro at 79.5% and Claude Sonnet 4.5 (Think) at 77.3%. Models released before September 2024 scored under 10% on HMMT 2025; performance rose with o1 and subsequent reasoning models. Spearman correlations above 80% between AIME, HMMT and CMIMC lead the authors to conclude that single-competition results "generalize well."
Comparing scores on the 2024 and 2025 editions against human performance quantiles, most models sit 10 to 20 points above the expected line on AIME, "suggesting inflated performance on AIME 2024 due to data contamination"; QwQ-Preview-32B exceeds it by nearly 60 points, "indicating extreme contamination." The discrepancy on HMMT is much smaller. A search using a deep-research tool found that "8 problems from AIME 2025 and 1 problem from HMMT 2025 can be found online in a similar form," mostly easier problems that do not affect the overall results.
On Project Euler with tools, GPT-5 scored 55%, Grok 4 and Grok 4 Fast 47.5%, o4-mini (high) 43.75%, Claude Sonnet 4.5 16.25% and Gemini-2.5-Pro 12.5%. On IMO 2025, six models were evaluated with best-of-32 selection in which the model judges a bracket tournament among its own proofs; GPT-5 scored highest at an average of 38% (16 points of 42), short of the 19 needed for a bronze medal, at about $200 for 24 answers. The authors recommend caution given the small number of problems.
Key claims
- Seven 2025 competitions, 162 problems, over 50 models; four runs per problem averaged; models released after a contest are flagged.
- Final-answer accuracy: GPT-5 91.3%, Grok 4 90.6%, Grok 4 Fast 90.4%, above the top 1% of human AIME and HMMT participants.
- AIME 2024 scores are inflated by 10 to 20 points relative to AIME 2025 for most models; QwQ-Preview-32B by nearly 60.
- IMO 2025 proofs (human-graded, best-of-32): GPT-5 38%, below the bronze threshold.
- Stated positions: final-answer benchmarks can be misleading; private benchmarks trade contamination resistance for transparency.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); the abstract page is arxiv.org. The USAMO 2025 procedure is detailed in the group's earlier report, Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025).
Relationships
- supports: MathArena — the primary text behind the platform's design and 2025 results.
- supports: AIME (as a model benchmark) — source of the AIME 2024 contamination finding and 2025 scores.
- related: Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025) — the same group's USAMO 2025 proof evaluation.
- contradicts: FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., Epoch AI, 2024) — argues that private benchmarks such as FrontierMath sacrifice reproducibility.
- related: AI Benchmarks and Evaluation