This is a research post published by FutureHouse on 23 July 2025, bylined to Andrew White with Michael Skarlinski, Jon Laurent and Albert Bou as co-authors, reporting an audit of the biology and chemistry questions in Humanity's Last Exam (HLE). It estimates that about 29% of the text-only chemistry and biology questions have answers contradicted by peer-reviewed literature, argues that the benchmark's construction incentives produced the problem, and releases a validated subset. An update dated 16 September 2025 records the HLE organizers' response.
Summary
Argument
The post describes HLE as "the most prominent eval representing PhD-level research," built by a consortium of subject-matter experts in response to saturation of benchmarks such as MMLU. It notes that as of July 2025 Grok 4 was the top-performing model at 26.9% accuracy without tools and 44% with tools and internet access. The authors' central claim is that HLE's inclusion criterion, under which a question was accepted only if frontier models could not answer it, "led to many 'gotcha' adversarial-type questions that ultimately confused the question writers and evaluators to the point that the questions lost all meaning."
The post illustrates this with an HLE question asking for the rarest noble gas on Earth as a percentage of terrestrial matter in 2002, whose keyed answer is oganesson. The authors call this "not a 'PhD level research' question, but instead trivia," and argue the answer is doubtful on three counts: only five atoms of oganesson have ever been made, it is predicted to be a solid rather than a gas, some chemists argue it is not noble because of its reactivity, and peer-reviewed tabulations of terrestrial noble-gas abundances do not include it. Two further examples are given: a pharmacy question on the beyond-use date of a punctured single-dose ampule, keyed as one hour, which the authors and an independent expert read USP <797> as answering "immediately" (the one-hour limit applying to vials rather than ampules); and an entomology question keyed to say that snakeflies (Raphidiopterans) feed on nectar as adults, for which the authors could find no source and cite literature recording predation and pollen but not nectar.
On process, the post states that HLE reviewers were instructed that questions "should ask for something precise and have an objectively correct, univocal answer," but were not required to verify a rationale if doing so would take more than five minutes, citing the HLE paper. The authors summarize: question writers had to verify that frontier models failed, reviewers had five minutes, and reviewers did not need to verify correctness.
Method
FutureHouse, which builds literature-research agents, ran its open-source PaperQA2 agent (called Crow) over the 321 text-only biology/health and chemistry questions in HLE, prompting it to find direct supporting or contradicting evidence for each question's answer and rationale and to label a pair "contradicted" only "if you are absolutely certain about the given answer being wrong." Crow labelled 53.3% (n=171) of the rationales as directly conflicting with published evidence, 57.0% in chemistry and 51.6% in biology/health. Independent expert chemistry and biology evaluators then examined 150 of the 321 outputs, rating Crow's judgment on a five-point scale from strongly disagree to strongly agree. The evaluators disagreed directly with Crow on 13.7% of the questions it had labelled contradicted and on 5.2% of those it had labelled correct.
Combining the agent's labels with the evaluators' corrections and extrapolating to the full subset, the authors estimate that 29.3% ± 3.7% (95% confidence interval) of the questions are "directly conflicted by research," 51.3% ± 4.1% are supported by research, and 19.3% ± 3.2% are "nuanced," meaning an assumption or opinion could change the answer. The post's headline figure is written as "29 ± 3.7%" in the abstract and "~29%" in the body. The full expert evaluation data are linked from the post.
Release and conclusion
FutureHouse released the question IDs its system graded as supported by research as HLE-Gold-Bio/Chem on Hugging Face (futurehouse/hle-gold-bio-chem) and recommends it for evaluating systems on HLE biology and chemistry, noting that HLE is a living dataset and that questions may change or be removed. The authors conclude that "the frontier of science isn't actually objective and univocal," which makes valid but challenging questions time-consuming to write, and express the hope that the analysis will inform the next generation of evaluations.
September 2025 update
The update states that the HLE team contacted FutureHouse after the post, ran an independent follow-up analysis and "found about 18% of a subset of questions in Bio/Chem were problematic," described in a revised section of the HLE preprint. The HLE team used a three-expert review, which FutureHouse calls "an excellent standard," and found that in 25% of cases at least one reviewer disagreed, "closer to our 29% number." FutureHouse reports that the HLE team planned to announce a rolling revision process.
Key claims
- Of 321 text-only biology/health and chemistry HLE questions, an estimated 29.3% ± 3.7% have answers directly contradicted by peer-reviewed literature, 51.3% ± 4.1% are supported and 19.3% ± 3.2% are nuanced.
- The PaperQA2 agent alone flagged 53.3% (171 questions) as contradicted; expert review of 150 outputs disagreed with the agent on 13.7% of contradicted labels and 5.2% of correct labels.
- The authors attribute the error rate to the adversarial inclusion rule and the five-minute limit on rationale verification.
- Grok 4 led HLE in July 2025 at 26.9% without tools and 44% with tools.
- The HLE organizers' follow-up review found about 18% problematic, or 25% under a single-dissent rule.
Provenance
Retrieved 22 September 2026 from futurehouse.org via firecrawl. The page carries a publication date of 23 July 2025 and an update dated 16 September 2025. The confidence rating reflects that the estimate rests on an agent-assisted audit with expert checking of a 150-item sample rather than full expert re-verification.
Relationships
- contradicts: Humanity's Last Exam (Phan et al., 2025; Nature, 2026) — the HLE authors' revised estimate for the same subset is about 18% under a multi-reviewer protocol, against 29.3% here.
- supports: Humanity's Last Exam — the source for the label-error estimates and the HLE Bio/Chem Gold subset described there.
- related: FutureHouse
- instance-of: AI Benchmarks and Evaluation