This is the paper introducing Humanity's Last Exam (HLE), a benchmark of 2,500 closed-ended academic questions organized by the Center for AI Safety and Scale AI. The organizing team of 22 lists Long Phan, Alice Gatti, Ziwen Han and Nathaniel Li as co-first authors and Summer Yue, Alexandr Wang and Dan Hendrycks as senior authors; the question contributors, listed by name, number in the hundreds. The paper was posted to arXiv as 2501.14249 in January 2025 and, after revision, published in Nature (volume 649, pages 1139–1146) on 28 January 2026 under the title "A benchmark of expert-level academic questions to assess AI capabilities," credited to the Center for AI Safety, Scale AI and the "HLE Contributors Consortium." The Nature version was received on 7 May 2025.
Summary
Motivation
The authors state that frontier language models "now achieve over 90% accuracy on popular benchmarks like MMLU," which they say limits the measurement of state-of-the-art capabilities. The arXiv version describes HLE as "designed to be the final closed-ended academic benchmark of its kind with broad subject coverage"; the Nature version replaces this with "designed to be an expert-level closed-ended academic benchmark with broad subject coverage." Both describe the questions as having "a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval," and both say HLE emphasizes "world-class mathematics problems."
Dataset and collection
HLE consists of 2,500 questions in more than a hundred subjects grouped into eight categories (mathematics, biology/medicine, physics, computer science/AI, humanities, chemistry, engineering, other). The questions came from "nearly 1000 subject expert contributors affiliated with over 500 institutions across 50 countries," mostly professors, researchers and graduate degree holders. About 24% of questions are multiple choice with five or more options and the rest require an exact-match answer; around 14% require an image as well as text. Each submission had to include the question, the answer specification, a detailed rationale, the subject, and the contributor's name and institution. Questions had to be original or non-trivial syntheses, precise, unambiguous, solvable and "non-searchable"; open-ended questions, subjective interpretation and content related to weapons of mass destruction were prohibited. Contributors were told that questions "typically require graduate-level expertise or test knowledge of highly specific topics (e.g., precise historical details, trivia, local customs)."
Submissions were encouraged with a $500,000 prize pool: $5,000 for each of the top 50 questions and $500 for each of the next 500, with co-authorship offered to anyone with an accepted question.
Review pipeline
Before a question could be submitted it was run against frontier models: GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet and o1 for all questions, with o1-mini and o1-preview added for text-only questions, and a GPT-4o judge scoring the responses. Exact-match questions had to defeat every model; multiple-choice questions had to defeat all but one, to allow for lucky guesses. The organizers logged more than 70,000 attempts, from which about 13,000 questions passed to human review. Reviewers holding graduate degrees graded submissions in their fields against standardized rubrics in two rounds: a first round of one to three reviews per question aimed at refining it, and a second round in which organizers and trained reviewers approved the final set. Reviewers "were not expected to verify the full accuracy of each provided solution rationale if it would take more than five minutes," and instead checked adherence to the guidelines.
Post-release refinement and expert disagreement
After release the organizers opened a bug-bounty programme for error reports, each verified by organizers with input from the original author; recruited students from US universities to solve a sample of questions in full; and audited "potentially searchable" questions (those a model answered correctly only with search tools, checked using GPT-4o mini and GPT-4o search and Perplexity Sonar), removing those easily found on the web. The authors report that model performance after this procedure was similar to before. Late contributions after release, each manually reviewed, formed a second held-out private set in addition to the original private set kept "to assess model overfitting and gaming on the public benchmark."
Two audit rounds of 200 questions each, with expert reviewers and an author rebuttal phase, produced "a final estimated expert disagreement rate of 15.4% for the public set." A targeted review of a biology, chemistry and health subset, prompted by an external analysis (About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025)), found a disagreement rate of about 18%, rising to 25% if a question is flagged whenever any single reviewer dissents, "which is close to the setting described in" that analysis. The authors argue that multiple reviewers and author rebuttal are needed because some answers rest on "a decades-old paper or a foundational concept not immediately apparent to others" or on contributors' unpublished research, and that some multiple-choice questions are designed to identify the most plausible option rather than an externally verifiable answer. They announce HLE-Rolling, "a dynamic fork of the dataset" to be "regularly updated to address community feedback and integrate new questions," as "a seamless migration path" for when models reach the ceiling on the original set.
Evaluation
Models were prompted with a fixed system prompt asking for an explanation, an answer and a confidence between 0% and 100%, and answers were judged against the key by o3-mini (o3-mini-2025-01-31 with structured decoding). Table 1 reports accuracy and RMS calibration error: GPT-4o 2.7% (89%), Grok 2 3.0% (87%), Claude 3.5 Sonnet 4.1% (84%), Gemini 1.5 Pro 4.6% (88%), Gemini 2.0 Flash Thinking 6.6% (82%), o1 8.0% (83%), DeepSeek-R1 8.5% (73%) and o3-mini (high) 13.4% (80%); the last two are text-only models evaluated on the text-only subset. On the text-only subset alone, GPT-4o scored 2.3%, Grok 2 3.2%, Claude 3.5 Sonnet 4.3%, Gemini 1.5 Pro 4.6%, Gemini 2.0 Flash Thinking 6.6% and o1 7.8%. The model versions were gpt-4o-2024-11-20, grok-2-latest, claude-3-5-sonnet-20241022, gemini-1.5-pro-002, gemini-2.0-flash-thinking-exp-01-21, o1-2024-12-17 and the 20 January 2025 DeepSeek-R1 release, at temperature 0 where configurable. Category scores for o3-mini (high) on text-only questions ranged from 18.6% in mathematics and 15.3% in physics to 5.2% in the humanities.
The authors caution that the low scores are "partially by design," that non-zero accuracy reflects noise and guessing on multiple-choice items, and that "small inflections close to zero accuracy are not strongly indicative of progress." They treat the calibration errors, all above 70%, as evidence that models "provide incorrect answers with high confidence." Reasoning models used far more completion tokens than non-reasoning models; the Nature version adds that accuracy rose log-linearly with output tokens up to about 2^14 tokens and then reversed.
Discussion
The arXiv version states that "it is plausible that models could exceed 50% accuracy on HLE by the end of 2025," and both versions say that high accuracy "would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or 'artificial general intelligence.'" The Nature version adds that the subject distribution is weighted toward mathematics and STEM and that HLE "is intended to hasten the transition towards a new class of benchmarks focused on more dynamic and open-ended AI capabilities." The dataset is released at huggingface.co/datasets/cais/hle and the inference code at github.com/centerforaisafety/hle.
Key claims
- 2,500 questions from nearly 1,000 contributors at more than 500 institutions in 50 countries; 24% multiple choice, about 14% with images.
- Questions were admitted only after defeating GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet and o1 (plus o1-mini and o1-preview for text-only); over 70,000 attempts yielded about 13,000 candidates for expert review.
- Release-time accuracy ranged from 2.7% (GPT-4o) to 13.4% (o3-mini high, text-only), with RMS calibration error above 70% for every model.
- Estimated expert disagreement of 15.4% on the public set and about 18% (25% under a single-dissent rule) on a biology, chemistry and health subset.
- Two private held-out sets are maintained; HLE-Rolling is the announced update path.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of the latest version of 2501.14249 (arxiv.org) and from the Nature article page (nature.com), both stored in one raw file. The Nature page's results table did not render in the capture; the figures above are from the arXiv version's Tables 1–4, which the Nature text references without contradiction.
Relationships
- supports: Humanity's Last Exam — the primary text for the benchmark's design, baseline results and error-rate estimates.
- contradicts: About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025) — the paper's 18% disagreement estimate for biology, chemistry and health is lower than FutureHouse's 29.3%, which the authors attribute to single-reviewer versus multi-reviewer methodology.
- depends-on: MMLU — the saturation of MMLU is the stated motivation.
- related: Center for AI Safety (CAIS), Scale AI, Dan Hendrycks
- instance-of: AI Benchmarks and Evaluation