MathArena

MathArena is an evaluation platform that runs language models on mathematics competitions as soon as their problems are released, maintained by the SRI Lab at ETH Zurich and INSAIT at Sofia University (Source: MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025); Source: matharena.ai). Its premise is that recurring competitions supply a stream of new, professionally vetted problems, and that a model evaluated before or just after a competition cannot have memorised it. It covers final-answer contests such as AIME and the HMMT tournament and proof-based olympiads such as the USA Mathematical Olympiad (USAMO) and the International Mathematical Olympiad (IMO), where human judges grade model proofs. Its 2025 USAMO study found that every model tested but one scored below 5% on proofs; one year later the best model scored 95% on USAMO 2026. In May 2026 the maintainers concluded that final-answer competition problems were no longer hard enough for frontier models and shifted to research-level benchmarks (Source: matharena.ai).

What it measures

The authors state three concerns about existing mathematics benchmarks: contamination of public test sets, the cost and opacity of private ones such as FrontierMath and Humanity's Last Exam, and an "emphasis on final answers" that lets a model succeed "through pattern recognition or brute-force enumeration, rather than genuine mathematical reasoning" MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025). MathArena is offered as a measure of mathematical problem solving on unseen problems and, in its proof tracks, of the ability to write a complete and rigorous argument. The authors describe it as "the first benchmark for proof-writing capabilities."

Design

Competition selection

Competitions are chosen for difficulty and reputation. The paper's calendar lists AIME, HMMT February, the BRUMO and CMIMC tournaments, USAMO, IMO and Project Euler as completed for 2025, with further tournaments and the Putnam planned; the authors tried other well-known contests such as Kangaroo and excluded them as already saturated. Problems are extracted from the official sources into a standard template and checked by hand. Competition organisers vet problems for originality, which the authors count as a contamination safeguard in itself MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Model selection and runs

A model is added if it competes for the top score, for the best open-weight score, or for the cost-performance frontier; non-reasoning models are excluded. Each model is run "once per competition using the hyperparameters recommended by the model providers, without further tuning," generating four responses per problem, and the reported score is the average. This is pass@1, accuracy with one attempt, estimated from four samples; no majority voting is used for final-answer contests. Project Euler runs allow code execution, capped at 20 executions per problem. The paper reports spending about $2,000 on API calls for its experiments MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Grading

Final answers are extracted by a custom rule-based parser into symbolic expressions and checked for equivalence; an LLM judge (Gemini 2.5 Flash) checks the same answers, and any disagreement is inspected by hand, because with 30 problems "even minor parser errors can have a disproportionate impact."

Proofs were, through 2025, graded only by people. Expert graders wrote a marking scheme intended to resemble the real competition's, with partial credit; solutions were anonymised; and two judges independently scored each out of 7 points MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025). For USAMO 2026 the team introduced a semi-automated pipeline built on juries of LLM judges, designed against two biases it identifies, self-bias (judges favouring their own model family) and formatting bias, with the resulting grades still "manually verified and adjusted" (Source: matharena.ai).

An example of a final-answer item, an HMMT problem shown in the paper: "Compute the sum of the positive divisors of 9! that have units digit 1." The answer is 103 MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Contamination controls

Evaluations are "conducted only a few hours to at most a few days after the competition concludes." A model released after a competition can still be added, but the leaderboard marks it with a warning sign. The authors acknowledge a residual risk that a closed model could be updated between a competition's release and its evaluation, which they consider minimal given training timescales MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Statistics and human baseline

Because competitions are small, the leaderboard reports a 95% confidence interval for each score and a rank interval from a paired permutation test. The paper also resampled four models 100 times per problem and found the four-run score estimates tightly distributed. Human reference points are the top 1% of contestants for AIME and HMMT and the medal cutoffs for the IMO (bronze, the top half of contestants, required 19 of 42 points in 2025) (Source: MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025); Source: matharena.ai).

Versions and variants

• Final-answer competitions: AIME 2025 and 2026, HMMT February 2025 and 2026, HMMT November 2025, BRUMO, SMT and CMIMC 2025. The site now labels this group "deprecated" (Source: matharena.ai).
• Apex and Apex Shortlist (August 2025): final-answer problems selected because no frontier model could solve them. By May 2026 GPT-5.5 had solved the last unsolved Apex problem, IMO 2025 Problem 6, and an attempt to build a second edition found no qualifying problems (Source: matharena.ai).
• Proof-based competitions: USAMO 2025 and 2026, IMO 2025, the International Mathematics Competition (IMC) 2025, Miklós Schweitzer 2025 and Putnam 2025 (Source: matharena.ai). The site labels this group "deprecated" as well and lists no IMO 2026 evaluation (Source: matharena.ai).
• Research-level benchmarks (2026): ArXivMath, final-answer questions derived from recent arXiv papers, issued in monthly editions; BrokenArXiv, which measures how often a model claims to prove a subtly false statement; ArXivLean, formal proofs in Lean 4 of statements from recent papers; and ProofRank, which rates aspects of proof quality beyond correctness (Source: matharena.ai).
• Visual mathematics: Math Kangaroo 2025, used to test vision-language models (Source: matharena.ai).

Development and maintainers

The paper's authors are Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović and Martin Vechev; the USAMO study adds Lyuben Baltadzhiev, Maria Drencheva and Kristian Minchev of INSAIT (Source: MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025); Source: Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025)). Code, problems and every model response are published, and the leaderboard lets a reader open any individual answer and grade. All scores are produced by the MathArena team; developers do not submit results. The USAMO 2026 post discloses partial support from a Google evaluation grant (Source: matharena.ai). The paper notes that the release notes for Phi-4-Reasoning, Gemini 2.5 Pro and Grok 3 cited MathArena results.

Results

On final-answer contests, the paper reports that models released before September 2024 scored under 10% on HMMT February 2025 and that by late 2025 GPT-5, Grok 4 and Grok 4 Fast averaged about 90–91% across AIME, HMMT, BRUMO and CMIMC, above the top 1% of human contestants MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025). On proofs, the March 2025 USAMO study found every model but one below 5% Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025); on IMO 2025 in July the best publicly available model reached 13 of 42 points, "not even bronze" in the post's title (Source: matharena.ai); on USAMO 2026 in March 2026 GPT-5.4 scored 95%, which the maintainers described as a "jump from near-zero performance to near-saturation in just one year" (Source: matharena.ai).

Separately from MathArena, AI developers have announced their own IMO results, collected in the last table below. In 2024 Google DeepMind's AlphaProof and AlphaGeometry 2 reached 28 points, silver-medal level, using formal languages and up to three days of computation. In 2025 an advanced version of Gemini Deep Think scored 35 of 42, the gold threshold, in natural language within the 4.5-hour limit and was "officially graded and certified by IMO coordinators"; OpenAI announced the same score for an experimental model two days earlier, graded by three former IMO medallists rather than by the IMO (Source: deepmind.google; Source: newscientist.com). In July 2026 Huawei and Xiaohongshu each said their systems had scored a full 42 (Source: france24.com).

HMMT February 2026

Final-answer contest. MathArena protocol: no tools, four runs per problem, pass@1. The site gives no run date per entry, so MathArena rows carry the date the leaderboard was read. Developer-reported rows are the developers' own runs on the same problems, not MathArena measurements.

USAMO 2026

Six proof problems, each out of 7 points, graded by an LLM-jury pipeline with manual verification; score is the percentage of 42 points, averaged over four runs. The accompanying post (Source: matharena.ai) gives the rounded figures 95%, 75%, 47% and 45% for the four models dated to it.

IMO 2025

Six proof problems graded by four human judges with IMO-level experience, two per solution, anonymised. Each model generated 32 proofs per problem and chose one itself through a bracket of pairwise comparisons (best-of-32); score is the percentage of 42 points, averaged over four runs.

USAMO 2025

Six proof problems graded by four human experts; four runs per model, no best-of-n selection. The first study, "Proof or Bluff?" Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025), was run within hours of the problems' release in March 2025 and reports the Gemini 2.5 Pro result as about 25%; later models were added afterwards. Figures are as listed on the leaderboard.

IMO results announced by developers

Points out of 42 on the year's six IMO problems. These are not MathArena measurements; the Setting column records who graded each result.

ArXivMath — June 2026, without tools

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArXivMath — June 2026, with tools

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArXivMath — March and April 2026 releases (71 problems)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArxivMath — June 2026 release

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArxivMath — April and May 2026 releases

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MathArena Apex — Apex

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Apex Shortlist

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HMMT Nov. 2025 — HMMT November 2025

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HMMT Feb. 2025 — HMMT February 2025

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

MathArena Apex 2025 — Apex

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArXivMath (MathArena) — August 2026, without tools

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArXivMath (MathArena) — August 2026, with tools

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

ArXivMath (MathArena) — August 2026, MathArena leaderboard setting

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Officially graded, self-graded and independently measured results are different kinds of claim. For IMO 2025, only DeepMind's result was marked by IMO coordinators. IMO president Gregor Dolinar confirmed that score and said the solutions were "clear, precise and most of them easy to follow," while the IMO's closing statement added that it "cannot validate the methods, including the amount of compute used or whether there was any human involvement, or whether the results can be reproduced" (Source: imo2025.au). Dolinar told New Scientist that OpenAI had asked about participating but did not register for the official grading scheme (Source: newscientist.com). The 35-point results came from unreleased systems with undisclosed compute; MathArena's 13 points for the best public model that month, and 16 for GPT-5 once it was released, measure what was available to users. Four of the 2026 perfect scores circulated in press coverage were graded by AI agents in a private test whose author wrote that the scores should be treated "as strong but not authoritative" (Source: github.com).

Selection inflates proof scores

MathArena's IMO 2025 figures use best-of-32 selection. One of its judges reviewed raw samples and estimated that without selection "model scores would likely have dropped below 10%" MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025). The USAMO 2025 figures use no selection, so the two proof tracks are not comparable.

Intervals are wide

With six problems, the 95% interval on a mid-range USAMO or IMO score is close to ±20 points, and the authors "recommend caution when interpreting the results, particularly when comparing models with similar scores." Final-answer tracks have intervals of 2 to 8 points.

Release date relative to the contest

A score carries MathArena's contamination guarantee only when the model predates the competition. Most current entries on the 2025 tracks, and many on the 2026 tracks, carry the warning sign.

Human terms

On final-answer contests, 2025's leading models exceeded the top 1% of contestants. On proofs, 19 of 42 was the bronze cutoff at IMO 2025 and 35 was a gold-medal score; at IMO 2026, per AFP, 7 of 666 contestants scored 42.

Limitations and criticism

Size and saturation

The authors state that the benchmark "remains small, leading to relatively wide confidence intervals," and predicted in the paper that answer-based contests "may soon become saturated, possibly as early as 2026." Their May 2026 post confirmed it: Gemini 3.1 Pro solved 162 of 176 candidate problems from recent competitions in all four attempts and the other 14 at least once, leading them to conclude that "final-answer competition problems are no longer an effective source of hard benchmark tasks for frontier models." They add that such problems remain useful for smaller models and academic research (Source: matharena.ai).

New problems are not always new

Using a deep-research tool, the team found that "8 problems from AIME 2025 and 1 problem from HMMT 2025 can be found online in a similar form" MathArena: Evaluating LLMs on Uncontaminated Math Competitions (Balunović et al., 2025).

Proof grading

Human grading is costly and cannot scale to every model. The USAMO 2025 study tested o3-mini and Claude 3.7 Sonnet as automatic graders and found that both "consistently overestimat[ed]" solution quality, "inflating the scores by a factor of up to 20" Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad (Petrov et al., 2025). The 2026 LLM-jury pipeline is the maintainers' response; its grades were still checked by hand, and the self-bias it guards against is the same concern raised about agent-graded IMO 2026 claims. The same study reported that, unlike human contestants, "all evaluated LLMs consistently claimed to have solved the problems," and that Gemini 2.5 Pro sometimes cited non-existent theorems.

Disputed third-party claims

MathArena's IMO post states that a paper claiming gold-medal performance for Gemini 2.5 Pro under an agentic framework had "several issues with the paper methodology," and records that re-running Grok 4 with a prompt xAI had used when training the model for proof generation raised its score from 11.90% to 21.43% (Source: matharena.ai).

Competition mathematics as a construct

Olympiad problems are timed puzzles with known solutions. The Fields Medallists' declaration of September 2026 argues that treating problem solving as the objective "may turn the tool against the primary goal" of understanding, and that results "announced in a rush" raise attribution problems; it names no benchmark or company and does not oppose the use of AI in mathematics (A Severe Misalignment of AI in Mathematics (Fields Medallists' declaration, 2026)). MathArena's own move toward research-level problems, false-statement detection and proof quality reflects a related judgement about what competition scores no longer show. For the separate question of how machine-produced research results are verified and credited, see Machine-produced mathematics: claims, verification and credit.

Use in policy and safety frameworks

MathArena results have been cited in developers' release notes (Phi-4-Reasoning, Gemini 2.5 Pro, Grok 3, per the paper), and its competitions (AIME, HMMT, USAMO) appear in launch tables from OpenAI, xAI, Z.ai and Thinking Machines Lab. The 2025 IMO results entered policy-facing reporting as a capability marker: the Stanford AI Index 2026 pairs "AI wins gold at the International Mathematical Olympiad" with failures on simple perception tasks to illustrate uneven capability (Stanford HAI AI Index Report 2026; Jagged Frontier). No use of MathArena in a capability threshold, regulation or standard was found.

Relationships

• instance-of: AI Benchmarks and Evaluation
• related: AIME (as a model benchmark) — the most widely reported competition MathArena covers; MathArena supplies its contamination analysis
• related: FrontierMath — the private-benchmark approach the authors criticise as hard to reproduce and selectively accessible
• related: MATH and MATH-500, GSM8K — earlier public benchmarks the authors describe as saturated and contaminated
• related: LiveCodeBench — a continuously refreshed benchmark built on the same contamination logic for code
• related: Google DeepMind, OpenAI — developers of the 2025 IMO gold-level systems
• related: Autoformalization and Machine-Checked Proof — ArXivLean and the Lean-based IMO systems
• related: A Severe Misalignment of AI in Mathematics (Fields Medallists' declaration, 2026), Machine-produced mathematics: claims, verification and credit

See also

• Jagged Frontier
• AI for Science
• Terence Tao