Humanity's Last Exam

Humanity's Last Exam (HLE) is a set of 2,500 closed-ended questions at the level of graduate study or specialist research, written by subject experts and kept only if the leading AI models of late 2024 answered them wrongly. It was organized by the Center for AI Safety and Scale AI, released in January 2025, and published in Nature on January 28, 2026 under the title "A benchmark of expert-level academic questions to assess AI capabilities" (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026)). The best model at release scored 13.4%. By September 2026 developers were reporting about 60% without tools and 65% with them, and Scale's own leaderboard had the leading model at 54.8% (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); anthropic.com; labs.scale.com). Audits of its answer key have put the share of sampled questions with disputed or defective answers between about 18%, in the maintainers' own review of a biology and chemistry subset, and 46%, in a 2026 sample by Epoch AI; the maintainers now publish a revised fork, HLE-Rolling.

What it measures

The authors present HLE as a response to the saturation of earlier tests: "LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities." They describe it as "designed to be the final closed-ended academic benchmark of its kind with broad subject coverage," in which each question "has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval." They also limit the claim: high accuracy "would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or 'artificial general intelligence,'" and "HLE may be the last academic exam we need to give to models, but it is far from the last benchmark for AI" Humanity's Last Exam (Phan et al., 2025; Nature, 2026). The paper states a policy purpose directly, saying that the benchmark "creates a common reference point for scientists and policymakers to assess AI capabilities."

Design

Dataset construction

Questions were crowdsourced from "nearly 1000 subject expert contributors affiliated with over 500 institutions across 50 countries – comprised mostly of professors, researchers, and graduate degree holders." Contributors competed for a $500,000 prize pool, with $5,000 for each of the top 50 questions and $500 for each of the next 500, and anyone with an accepted question was offered co-authorship of the paper. Each submission had to include the question, the answer, a detailed rationale, the subject, and the contributor's name and institution. Questions had to be original, precise, unambiguous and "non-searchable"; open-ended questions, subjective interpretation and content related to weapons of mass destruction were excluded Humanity's Last Exam (Phan et al., 2025; Nature, 2026).

The distinctive step was an adversarial filter against frontier models. Before a question could be submitted it was run against GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet and o1, plus o1-mini and o1-preview for text-only questions. An exact-answer question had to defeat every model; a multiple-choice question had to defeat all but one, to allow for lucky guesses. The organizers logged more than 70,000 attempts, of which about 13,000 questions passed and went to human review. Reviewers with graduate degrees, recruited through Scale's Outlier platform, scored questions against a rubric in a first round of one to three reviews each, and organizers and trained reviewers approved the final set in a second round. Reviewers "were not expected to verify the full accuracy of each provided solution rationale if it would take more than five minutes" (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); labs.scale.com).

After release the organizers ran a bug-bounty programme for error reports, which closed on March 21, 2025; recruited university students to solve a sample of questions in full; and removed questions that models with web search answered correctly and that proved easy to find online. The dataset was finalized at 2,500 questions on April 3, 2025, with removed questions partly replaced from a reserve pool, and the earlier version is referred to as HLE-preview (Source: labs.scale.com; Humanity's Last Exam (Phan et al., 2025; Nature, 2026)).

Task format and an example

About 24% of questions are multiple choice with five or more options; the rest require an exact short answer such as a number, a name or a formula. About 14% include an image that must be understood together with the text. The most common subjects include mathematics, biology and medicine, physics, computer science, the humanities, chemistry and engineering, within a list of more than a hundred that also covers fields such as classics, musicology, chess and classical ballet. The authors say questions "typically require graduate-level expertise or test knowledge of highly specific topics (e.g., precise historical details, trivia, local customs)" Humanity's Last Exam (Phan et al., 2025; Nature, 2026). One chemistry question quoted by critics reads: "What was the rarest noble gas on Earth as as a percentage of all terrestrial matter in 2002?", with the keyed answer "Oganesson" About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025).

Scoring and calibration

A model is given a standard system prompt asking for its reasoning, a final answer and a confidence from 0% to 100%. Because exact answers can be written in equivalent forms, a judge model (OpenAI's o3-mini in the paper and on Scale's leaderboard) compares each response with the answer key. The accuracy score is the percentage judged correct. The maintainers note that "small differences could arise from different judge models and prompts used on edge cases" and ask evaluators to document theirs (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); labs.scale.com).

The second metric is RMS calibration error, which measures the gap between a model's stated confidence and its actual accuracy: a well-calibrated model that says it is 50% confident should be right half the time. A high calibration error with low accuracy means the model is confidently wrong, which the authors treat as evidence of hallucination. All models in the paper had calibration errors above 70% Humanity's Last Exam (Phan et al., 2025; Nature, 2026).

Human baseline, splits and contamination controls

No human baseline is reported; the questions are each written by a specialist and no one is expected to answer across all fields. The 2,500 questions are public, distributed through a gated Hugging Face repository with a request not to re-share them. The organizers hold a private held-out set "to assess model overfitting and gaming on the public benchmark," and a second held-out set assembled from late submissions; Scale states that it does not currently report scores on the private set (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); labs.scale.com; huggingface.co).

Versions and variants

• HLE-preview is the pre-April 2025 set of roughly 3,000 questions; OpenAI's February 2025 deep research result was measured on "over 3,000" questions (Source: openai.com).
• The text-only subset excludes the image questions. It is used for models that cannot read images and, by some developers, as their default; Z.ai states that it reports the text-only subset unless a figure is marked as full-set (Source: z.ai). Moonshot AI labels its full-set figures "HLE-Full" (Kimi K3: Open Frontier Intelligence (Moonshot AI, July 2026)).
• With tools results allow web search and code execution. Developers usually block access to pages that contain the answers; Google DeepMind describes its setting as "Search (blocklist) + Code" (Source: deepmind.google).
• HLE Bio/Chem Gold is a subset of biology and chemistry questions that FutureHouse judged to be supported by the literature, released on Hugging Face in July 2025 About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025).
• HLE-Verified (Qwen Team, Alibaba, February 2026) re-examined all 2,500 items, certifying 668 as correct, repairing and certifying 1,143, and releasing the remaining 689 as a documented uncertain set. Eight models scored 7–10 points higher on it than on the original, and 30–40 points higher on items whose question or answer had been wrong (Source: arxiv.org). Google reported 54.9% on HLE-Verified for Gemini 3.8 Flash in September 2026 (Gemini 3 / Gemini 3 Pro).
• HLE-Rolling is the maintainers' own updated fork, announced in the revised paper as "regularly updated to address community feedback and integrate new questions." Scale announced a September 17, 2026 version that also "replaced some easy questions with harder questions from our held out set," and describes the fork as "a seamless migration path" for when models "begin to hit the noise ceiling on the original HLE dataset" (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); labs.scale.com).

Development and maintainers

The organizing team lists 22 people from the Center for AI Safety and Scale AI, with Summer Yue, Alexandr Wang and Dan Hendrycks as senior authors; Hendrycks is also first author of MMLU, the benchmark whose saturation the paper cites as its motivation. The Nature version states that the two organizations "jointly designed the dataset premise and pipeline" and operated the collection platform (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026)).

Results are published in three ways. Scale's SEAL leaderboard evaluates models itself at temperature 0 with the o3-mini judge, reports a 95% confidence interval and the calibration error for each, and ranks models by whether their intervals overlap. Scale is both co-creator and evaluator. Developers report their own figures in launch tables, usually in a no-tools and a with-tools setting. Artificial Analysis runs HLE independently as one of the ten components of its Intelligence Index (Source: labs.scale.com; artificialanalysis.ai).

Results

In the paper's pre-release evaluation GPT-4o scored 2.7%, Claude 3.5 Sonnet 4.1%, o1 8.0% and o3-mini (high) 13.4%, the last on the text-only subset. The authors wrote that "it is plausible that models could exceed 50% accuracy on HLE by the end of 2025" Humanity's Last Exam (Phan et al., 2025; Nature, 2026). With tools that forecast was met within the year: OpenAI's deep research agent reached 26.6% in February 2025 with browsing and Python, xAI reported 50.7% for Grok 4 Heavy on the text-only subset in July 2025, and Anthropic reported 53.0% for Claude Opus 4.6 with tools in February 2026 (Source: openai.com; x.ai; anthropic.com). Without tools, reported scores rose from 18.8% for Gemini 2.5 Pro in March 2025 to 37.5% for Gemini 3 Pro in November 2025, 46.9% for Claude Opus 4.7 in April 2026 and 60.9% for Claude Fable 5.1 in September 2026 (Source: blog.google; blog.google; openai.com; anthropic.com). Scale's own measurements run lower than developers' for the same models, with GPT-6 Astra highest at 54.80% (Source: labs.scale.com).

No tools

Accuracy on the full 2,500-question set, including image questions, with no search or code execution. Rows measured by Scale AI use temperature 0 and an o3-mini judge, and give Scale's 95% confidence interval and RMS calibration error; the Date for those rows is the model's release date, since the leaderboard does not date entries.

With tools

Accuracy with web search and code execution available. Tool sets, blocklists and context management differ between developers, and some rows are on the text-only subset, as marked.

Text-only subset, no tools

Accuracy on the questions without images, with no tools.

Humanity's Last Exam — RMS calibration error

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Humanity's Last Exam — Calibration AUROC

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HLE Text — AA (Artificial Analysis harness)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HLE-Verified

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Low scores were built in

Every question was selected because late-2024 frontier models failed it, so those models' scores near zero reflect the selection rule as well as difficulty; the authors write that "small inflections close to zero accuracy are not strongly indicative of progress." Because a quarter of the questions are multiple choice, a model guessing at random would still score a few percent; Epoch AI's benchmark metadata puts the chance level at 4.8% (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); epoch.ai).

The ceiling is well below 100%

The maintainers' own estimate is that experts disagree with the answer key on 15.4% of public questions, and 18% in biology, chemistry and health Humanity's Last Exam (Phan et al., 2025; Nature, 2026). Outside audits have produced higher figures, described under Limitations. Scale refers to a "noise ceiling" that frontier models will reach on the original set, and the HLE-Verified authors found that models' scores rise by 7–10 points once erroneous items are repaired, which implies that part of the remaining gap to 100% is error in the key rather than missing capability (Source: labs.scale.com; arxiv.org).

Tools, subset and grader each move the number

For the same model, tools add roughly 4 to 16 points in the tables above. Developer-reported and Scale-measured figures for the same model differ in both directions: Claude Opus 4.7 appears at 46.9% in OpenAI's table and 36.20% on Scale's leaderboard, while GPT-5.4 Pro appears at 42.7% from OpenAI and 44.32% from Scale. Anthropic revised its own Sonnet 4.6 figures in June 2026 "after a grader-model change," and Z.ai used a different judge model for GLM-5.3 than the o3-mini judge Scale uses (Claude Sonnet 5; Source: z.ai). Anthropic notes that its Fable 5.1 figures were obtained with production safeguards enabled, which it says likely depresses them (Claude Fable 5.1).

Calibration has improved less evenly than accuracy

On Scale's leaderboard calibration errors range from 20% for Claude Fable 5.1 to 51% for Gemini 3.1 Pro among models with similar accuracy, against 73–89% for the models in the paper (Source: labs.scale.com). Meta's safety report for Muse Spark states that "all models exhibit poor confidence calibration when measured on Humanity's Last Exam," giving its own model an RMS error of 50.3 (Muse Spark Safety & Preparedness Report (Meta, May 2026)).

What a score stands for

The authors' own reading is that a high score shows "expert-level performance on closed-ended, verifiable questions," not research ability. Critics add that many items test obscure recall rather than reasoning; FutureHouse called the oganesson item "not a 'PhD level research' question, but instead trivia" About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025).

Limitations and criticism

FutureHouse audit (July 2025)

FutureHouse, which builds literature-research agents, ran its PaperQA2-based agent over the 321 text-only biology, health and chemistry questions, asking it to find evidence for or against each official rationale, and had independent experts check 150 of the outputs. It estimated that 29.3% ± 3.7% of those questions have answers directly contradicted by peer-reviewed literature, 51.3% are supported and 19.3% are "nuanced." It attributed the problem to the benchmark's incentives: writers had to defeat frontier models, which encouraged "'gotcha' adversarial-type questions," while reviewers were not required to verify rationales that would take more than five minutes About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025).

The maintainers' response

According to FutureHouse's September 2025 update, the HLE team contacted it and ran an independent three-expert review of a biology and chemistry subset. The revised paper reports an expert disagreement rate of about 18% on that subset, rising to 25% if a question is flagged whenever any single reviewer dissents, "which is close to the setting described in" the FutureHouse post. The authors argue that multiple reviewers and an author rebuttal are needed because some answers rest on "a decades-old paper or a foundational concept not immediately apparent to others" or on contributors' own unpublished research experience, and they announced HLE-Rolling as the vehicle for continuing corrections (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); About 30% of Humanity's Last Exam Chemistry/Biology Answers are Likely Wrong (FutureHouse, 2025)).

Epoch AI review (August 2026)

Epoch AI audited a random sample of 48 questions, six from each of eight categories, using Claude Fable 5 to surface candidate errors that its staff then confirmed by hand. It found 12 questions impossible to answer correctly as written and 10 more whose key could mark a correct answer wrong, 22 of 48 or 46% in all, and gave the original benchmark the verdict "Flawed." Its examples include a question asking for a "4-point" discrete Fourier transform of an eight-entry sequence and an answer key that accepts only one of two common abbreviations for the biblical book 1 Kings. The review covered the original dataset, not HLE-Rolling or HLE-Verified (Source: epoch.ai).

HLE-Verified (February 2026)

The Qwen Team's item-by-item verification left 689 of 2,500 items (28%) in an "uncertain" set and found it necessary to repair 1,143 (46%) (Source: arxiv.org).

Contamination and searchability

The public questions and answers can enter training data, and with-tools runs can find them online. The maintainers' controls are the private held-out sets, the gated dataset, and the 2025 removal of searchable questions; developers apply their own blocklists (Source: Humanity's Last Exam (Phan et al., 2025; Nature, 2026); deepmind.google).

Construct

The Stanford AI Index 2025 listed HLE among the harder evaluations researchers turned to as older benchmarks saturated (Source: hai.stanford.edu). Ethan Mollick cites HLE with GPQA and GDPval as showing the same exponential improvement curve (The Shape of the Thing), while accounts of the jagged frontier group HLE with ARC-AGI, FrontierMath and GPQA as successor benchmarks that each expose a different facet of uneven model capability.

Use in policy and safety frameworks

HLE is a capability benchmark and is not used as a threshold in frontier safety frameworks. It appears in developers' launch tables and system cards as a headline measure of expert knowledge, and in safety documentation mainly through its calibration metric: Meta's Muse Spark safety and preparedness report uses HLE to assess whether the model knows when it does not know. OpenAI's CoT-Control suite, which tests whether a model can be instructed to manipulate its visible reasoning and is reported in the GPT-6 Astra system card, draws part of its 13,000 tasks from HLE (Reasoning models struggle to control their chains of thought, and that's good (OpenAI, March 2026)). Researchers studying whether open-weight models were trained on proprietary models' reasoning traces used 30 HLE problems as probes (Stealing Reasoning Traces from Proprietary LLM APIs). HLE is one of three benchmarks averaged in the "Reasoning" row of Thomson Reuters's comparison of its legal model with general models (Thomson Reuters Built Its Own AI Model That Now Ranks Among the World's Best), and it is a component of the Artificial Analysis Intelligence Index (Source: artificialanalysis.ai).

Relationships

• instance-of: AI Benchmarks and Evaluation
• depends-on: Center for AI Safety (CAIS), Scale AI — co-creators and maintainers.
• related: Dan Hendrycks — senior author.
• supersedes: MMLU — the saturated benchmark the authors cite as motivation, from the same senior author.
• related: GPQA — the earlier expert-written, search-resistant question set.
• related: Artificial Analysis Intelligence Index, Epoch Capabilities Index (ECI) — composite indices that draw on HLE results.
• depends-on: SimpleQA — the SimpleQA paper (Wei et al.) is the source of the confidence-elicitation setup used for the calibration metric.

See also

• FrontierMath
• BrowseComp — the usual companion benchmark for with-tools search settings.
• LAB-Bench
• Stanford HAI AI Index Report 2026