"Is AI already superhuman on FrontierMath?" is a post by Anson Ho, a senior researcher at Epoch AI, published on 23 May 2025 in Epoch's Gradient Updates newsletter, which the organisation describes as carrying "more opinionated or informal takes" that "solely represent the views of the authors." It reports the results of a human-baseline competition on FrontierMath held at MIT and discusses how a human baseline for the benchmark should be defined.
Summary
Ho describes FrontierMath as containing 300 questions ranging from upper-undergraduate level to problems "that even Fields Medallists find challenging." To establish a human baseline, Epoch organised a competition at MIT with "around forty exceptional math undergrads and subject matter experts," split into eight teams of four or five and given 4.5 hours to solve 23 questions with internet access, code in any language, and no generative AI tools or outside contact. The comparison model was o4-mini-medium, "the current state-of-the-art AI system on FrontierMath." The result: o4-mini-medium scored 22%, above the average team's 19% but below the 35% of problems solved by at least one team, and it "only managed to solve problems that at least one human team had solved." Ho's conclusion is that "AIs aren't yet unambiguously superhuman on FrontierMath – but I think they soon will be." Gemini 2.5 Pro solved exactly the same 5 of 23 problems as o4-mini-medium in preliminary evaluations.
The post then sets out four qualifications.
Subject-matter expertise was underrepresented. Participants needed exceptional ability (PhDs or strong competition records), and each team had at least one expert in some domain, but "most teams had a roughly even split between undergrads and experts," and none had experts in all four advanced domains. Participants were largely from Boston's mathematics community, "which famously has a limited representation of domain expertise in analysis."
The competition was designed to reflect reasoning rather than broad knowledge. Since no human can cover the full range of fields in FrontierMath, problems were chosen to need less background or to match participants' expertise: seven "general" questions accessible to strong undergraduates (originally eight, one removed because its author was a participant) and sixteen "advanced" questions in topology, algebraic geometry, combinatorics and number theory. Advanced problems counted two points against one for general ones, with a bonus point per domain with at least one correct answer. Prizes were $1,000, $800 and $400 for the top three teams and $150 honoraria for others. Ho notes this gives humans an advantage and that on the full benchmark, if reasoning is not the specific concern, "it's very likely that AIs are already superhuman."
The definition of "human baseline" is ambiguous. Teams solved between 13% and 26%, averaging 19%, which Ho calls "roughly on par" with o4-mini-medium's 22% given the sample sizes. Counting a problem as solved if any team solved it gives about 35%, which he considers "a bit too optimistic" since it resembles pass@8 against the model's pass@1; the competition-level human baseline is "probably somewhere in between," around 20–30%. The competition's difficulty mix (17% Tier 1, 13% Tier 2, 70% Tier 3) also differs from the full benchmark's (20%, 40%, 40%); reweighting raises the human baseline to about 30% by team average and 52% by any-team, but applying the same reweighting would put o4-mini-medium at about 37% against the 19% Epoch measured on the full benchmark, so Ho is "not sure that this method of doing the adjustment is really any good." A footnote reports that on the 8 questions rated undergraduate level or below, o4-mini-medium scored around 50%, similar to the team average, with 75% by the any-team method.
Models are not yet superhuman but probably soon will be. Ho writes that if the relevant baseline is around 30–50%, "it's likely that AIs will unambiguously exceed this threshold by the end of the year." He cautions that human performance may be understated by the format: o4-mini-medium took 5–20 minutes per problem, while survey respondents spent a mean of about 40 minutes on their favourite problem, and related work on machine-learning tasks suggests humans keep improving with time where models plateau. He also notes that FrontierMath questions "aren't a direct representation of actual math research." The competition was funded by Open Philanthropy.
Key claims
- 23 FrontierMath problems, eight teams of four or five, 4.5 hours, internet access; average team 19% (range 13–26%), any-team 35%, o4-mini-medium 22%.
- o4-mini-medium's full-benchmark score at the time was 19% (Epoch's own evaluation).
- Ho's stated view: the most informative human baseline is between 30% and 50%, with the adjustment methods he tried described as unreliable.
- Ho's forecast, May 2025: models will unambiguously exceed that baseline by the end of 2025.
Provenance
Retrieved 22 September 2026 from epoch.ai. Competition results are linked from the post as a spreadsheet; the competition itself was announced separately on Epoch's blog.
Relationships
- supports: FrontierMath — source of the benchmark's human-baseline figures.
- depends-on: FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI (Glazer et al., Epoch AI, 2024) — the benchmark paper.
- related: Announcing FrontierMath Erdős (Epoch AI, September 2026) — Epoch's later extension of the benchmark.
- related: Epoch AI, AI Benchmarks and Evaluation