This is the paper by Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke and Karan Singhal of OpenAI that introduces HealthBench, an open-source benchmark of health conversations graded by physician-written rubrics. It was submitted to arXiv as 2505.08775 on 13 May 2025, with Singhal as corresponding author. The code and data are released through OpenAI's simple-evals repository.
Summary
Motivation
The authors argue that existing health evaluations fall short on three dimensions: they are not meaningful because multiple-choice exams and narrow clinical questions are poorly aligned with open-ended real situations; not trustworthy because they lack validation against expert medical opinion; and not unsaturated because many leave no room for improvement. HealthBench is presented as addressing all three.
Rubric evaluation
Each of the 5,000 examples is a conversation, single- or multi-turn, ending with a user message, paired with rubric criteria written by a physician for that conversation. Criteria describe what a response should include or avoid, from specific facts and dosages to asking the user for more detail, and each carries a nonzero point value between −10 and 10, with negative values for undesirable behaviour. A model-based grader decides independently for each criterion whether the response meets it; met criteria contribute their full points, the total is divided by the maximum possible score, and a per-example score can be negative. The overall score is the mean of per-example scores clipped to [0, 1]. Examples have a mean of 2.6 turns (range 1–19), a mean length of 668 characters (range 4–9,853), and a median of 11 criteria (range 2–48). There are 48,562 unique criteria; counting each assignment, 57,237 criteria are partitioned into five axes: completeness (39%), accuracy (33%), context awareness (16%), communication quality (8%) and instruction following (4%).
Examples are partitioned into seven themes: global health (1,097 examples, 21.9%), responding under uncertainty (1,071, 21.4%), expertise-tailored communication (919, 18.4%), context seeking (594, 11.9%), emergency referrals (482, 9.6%), health data tasks (477, 9.5%) and response depth (360, 7.2%).
Data collection
The tasks were created over 11 months by 262 physicians with practice experience across 60 countries and 26 specialties, who between them spoke 49 languages well enough to contribute data. They were selected from 1,021 who expressed interest (26%) on the basis of an interest form and paid introductory tasks including rubric writing; a further 31 were filtered out during the campaign and their annotations removed. Half were independent-practice or staff physicians, 17% fellows, 23% residents in their third year or above and 10% first- or second-year residents. Most conversations were generated synthetically from physician-described situation types, with others drawn from physician red-teaming and from Google's HealthSearchQA queries rewritten by a language model into conversations.
Variants
Thirty-four pre-written consensus criteria (8,053 assignments) are attached to a conversation only when two or more reviewing physicians agree they apply, for example whether a reply to a likely emergency tells the user to seek immediate care. HealthBench Consensus keeps the 3,671 examples with at least one positive consensus criterion and scores only those criteria; the authors describe it as having greater precision but lower recall in finding failures, and report results as error rates. HealthBench Hard is a subset of 1,000 examples chosen for difficulty for frontier models, on which many models score zero.
Results
Across 16 repeated runs at temperature 1.0, mean scores were 0.5990 for o3, 0.4778 for GPT-4.1, 0.4200 for o1, 0.3233 for GPT-4o (August 2024) and 0.1554 for GPT-3.5 Turbo, with standard deviations of about 0.002. The authors say OpenAI's frontier models improved by 28% "in recent months," a larger step than from GPT-3.5 Turbo to GPT-4o, and that GPT-4.1 nano outperforms GPT-4o while being 25 times cheaper. On HealthBench Consensus, error rates fell by over four times from GPT-3.5 through GPT-4.1, with context-seeking behaviour, responding under uncertainty and response depth identified as areas of remaining weakness; the per-criterion table shows, for example, GPT-4.1 meeting the context-seeking criterion on only 18.8% of conversations that lacked sufficient context. On HealthBench Hard, o3 scored 0.32 against 0.60 overall.
A worst-at-k analysis takes the worst of k sampled responses per problem; o3's worst-at-16 score was reduced by a third relative to its overall score, and was more than double that of GPT-4o. Within-model correlation between response length and score was small (Pearson's r = −0.053 for GPT-4o, 0.123 for o3); o3's win rate against GPT-4.1 fell from 72.9% to 63.7% when only response pairs within 10% of each other's length were compared.
Physician-written responses
Physicians were asked to write ideal responses in three conditions. Those writing without AI assistance scored below the models. Physicians given reference responses from the September 2024 models (GPT-4o and o1-preview) improved on them more often than they worsened them (56.2% against 39.8%), whereas those given responses from the April 2025 models (GPT-4.1 and o3) were equally likely to improve or worsen them (46.8% against 47.7%).
Trustworthiness of grading
Meta-evaluation used the consensus criteria, for which physicians also graded whether responses met each criterion, giving over 60,896 meta-examples (average 1,791 per criterion). Macro F1 between model and physician grades was 0.709 for GPT-4.1, 0.692 for o4-mini, 0.681 for o3, 0.661 for GPT-4.1 mini and 0.580 for GPT-4.1 nano; the authors report that model-physician agreement is similar to physician-physician agreement and adopt GPT-4.1 as the default grader.
Key claims
- 5,000 conversations; 48,562 unique rubric criteria; 262 physicians from 60 countries and 26 specialties; seven themes and five axes.
- Scores: o3 59.9%, GPT-4.1 47.8%, o1 42.0%, GPT-4o 32.3%, GPT-3.5 Turbo 15.5% (mean of 16 runs).
- HealthBench Hard (1,000 examples): top score 32% (o3). HealthBench Consensus: 3,671 examples, 34 criteria.
- Grader GPT-4.1 macro F1 0.709 against physician labels on 60,896 meta-examples.
- Unassisted physicians scored below the models; physicians could improve September 2024 model responses but not April 2025 ones.
- The paper states the benchmark does not measure health outcomes of specific workflows.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org); abstract page arxiv.org. Code and data: github.com.
Relationships
- supports: HealthBench — the primary text for the benchmark's design, variants and baseline scores.
- related: HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats (Hicks et al., OpenAI, 2026) — the April 2026 clinician-focused successor from the same group.
- instance-of: AI Benchmarks and Evaluation
- related: AI in Healthcare, Healthcare — AI Deployment, OpenAI