HealthBench

HealthBench is OpenAI's evaluation for health-related conversations. OpenAI released it as an open-source benchmark on May 12, 2025 (Source: openai.com; HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025)) and uses it as the reported measure of performance for its ChatGPT Health product (Source: openai.com). A model is shown a conversation between a user, who may be a member of the public or a clinician, and an assistant, and must write the next reply. A grader model then checks the reply against a rubric that a physician wrote for that specific conversation. OpenAI's o3 scored 60% at release. By 2026 OpenAI's system cards described the original benchmark as approaching a noise ceiling for frontier models and recommended HealthBench Professional, built from real clinician chats, for tracking further progress (GPT-5.6 Preview System Card (OpenAI, June 2026)).

What it measures

The design choice that distinguishes HealthBench from earlier medical AI benchmarks is what it declines to measure. Rather than exam-style accuracy, the USMLE-question format on which language models had already reached high scores, it uses physician-written rubrics that, in OpenAI's words, reflect "how clinicians judge quality in practice—prioritizing safety, clarity, appropriate escalation of care, and respect for individual context" (Source: openai.com). The paper frames the goal as an evaluation that is meaningful (scores reflect real-world impact), trustworthy (scores track physician judgment) and unsaturated HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Escalation appropriateness scores whether the system directs a user to clinical care when it should, which is a property of the interaction and not only of the model's knowledge. The paper's emergency-referrals theme treats errors in both directions as harmful: failing to escalate when needed, and over-escalating in ways that strain health systems. A related theme, context-seeking, tests whether a model recognizes that key information is missing and asks for it before answering HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Design

Physician cohort

The benchmark was produced over 11 months with 262 physicians who had practiced in 60 countries across 26 specialties and who between them could contribute in 49 languages. They were selected from 1,021 who expressed interest, were paid, and were monitored for quality; a further 31 were filtered out during the project and their annotations removed. Half were independent-practice or staff physicians and the rest fellows and residents HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025). OpenAI's later description of its wider physician program, on the ChatGPT Health launch page, says that over two years it worked with more than 260 physicians who have practiced in 60 countries and dozens of specialties, and that this group provided feedback on model outputs more than 600,000 times across 30 areas of focus (Source: openai.com). The breadth across countries and specialties addresses a standing criticism of medical AI benchmarks, that they encode the norms of a single health system; the paper's global-health theme tests whether a model adapts to differences in resources, clinical norms and disease patterns HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Conversations

There are 5,000 conversations, each ending with a user message, averaging 2.6 turns and ranging from one to nineteen. Most were generated synthetically: physicians listed the kinds of situation that mattered, for example "a healthcare professional asking a model to summarize a clinical note" or a user whose details over several turns point to an emergency, and a language-model pipeline turned these into realistic conversations. The rest came from physicians' red-teaming of models and from rewritten queries in Google's HealthSearchQA dataset. A model (o1-preview) filtered conversations for realism and relevance. The conversations fall into seven themes: global health (21.9%), responding under uncertainty (21.4%), expertise-tailored communication (18.4%), context seeking (11.9%), emergency referrals (9.6%), health data tasks (9.5%) and response depth (7.2%) HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025). A commentary in Digital Health counts 4,248 prompts (85.0%) in English, 192 in Spanish, 173 in Portuguese and 387 across 29 other languages (Source: pmc.ncbi.nlm.nih.gov).

Rubrics and scoring

For each conversation a physician wrote criteria describing what a good reply should or should not contain, such as a specific fact, a recommended dosage or a request for more detail about the user's symptoms, each with a point value between -10 and 10. The median conversation has 11 criteria and the dataset has 48,562 unique ones, classified on five axes: completeness (39%), accuracy (33%), context awareness (16%), communication quality (8%) and instruction following (4%). Apart from the consensus criteria described below, criteria were written by one physician and not validated by others. The grader model decides independently for each criterion whether the reply meets it. A conversation's score is the points earned, including penalties, divided by the maximum positive points, so it can be negative; the benchmark score is the mean across conversations, clipped to between 0 and 1. The rubrics, data and grading code were released in OpenAI's simple-evals repository HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Validation of the grader

Thirty-four pre-written consensus criteria, for example whether a reply to a likely emergency tells the user to seek immediate care in its first sentences, are attached to a conversation only when two or more physicians agree they apply. Physicians also graded model replies on these criteria, giving 60,896 comparisons between physician and model grading. GPT-4.1 as grader reached a macro F1 of 0.709, above the average physician in five of seven themes, and was adopted as the default grader; agreement between physicians, and between model and physicians, ranged from 55% to 75%. Across 16 repeated runs the overall score had a standard deviation of about 0.002 HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Physician baseline

Physicians were asked to write ideal replies under three conditions. Those working without AI help scored below the models, which the paper cautions reflects partly that writing chat replies is not a normal physician task and that their replies were shorter. Physicians given replies from the September 2024 models (GPT-4o and o1-preview) improved on them more often than they worsened them (56.2% against 39.8%); physicians given replies from the April 2025 models (GPT-4.1 and o3) were as likely to worsen as improve them (46.8% against 47.7%) HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Versions and variants

• HealthBench Hard is a subset of 1,000 conversations chosen because frontier models of early 2025 did poorly on them; many models scored zero and o3 scored 32%.
• HealthBench Consensus keeps the 3,671 conversations with at least one consensus criterion and scores only those criteria. The paper describes it as more precise but narrower; results are usually reported above 90%, and the paper discusses them as error rates HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).
• Length adjustment. Because longer replies have more chances to satisfy criteria, OpenAI's 2026 system cards report scores adjusted for reply length: a reply of 2,000 characters is unadjusted, and each additional 500 characters costs 2.99 points on HealthBench, 3.92 on HealthBench Hard, 0.20 on Consensus and 1.47 on HealthBench Professional, with shorter replies credited correspondingly (GPT-5.6 Preview System Card (OpenAI, June 2026)).
• HealthBench Professional (April 2026) has 525 tasks taken from conversations physicians held with OpenAI's ChatGPT for Clinicians product, in three use cases: care consultation, writing and documentation, and medical research. Every rubric was written and adjudicated by three or more physicians, from a cohort of 190. About a third of tasks come from deliberate adversarial testing, and the final set was drawn from 15,079 candidates with difficult examples over-sampled about 3.5 times, so the authors state that absolute scores are not comparable to real-world performance rates. The grader is GPT-5.4 and the reported score is length-adjusted. Specialty-matched physicians writing with web access and no time limit scored 43.7, base GPT-5.4 48.1 and GPT-5.4 inside ChatGPT for Clinicians 59.0 HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats (Hicks et al., OpenAI, 2026).

Development and maintainers

The HealthBench paper's authors are Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke and Karan Singhal, all of OpenAI; Hicks, Mikhail Trofimov and Singhal are the corresponding authors of the HealthBench Professional paper (Source: HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025); HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats (Hicks et al., OpenAI, 2026)). There is no central leaderboard and no verification of submitted results. Developers run the open-source evaluation themselves and may choose their own grader: Anthropic's system cards report using Claude Opus 4.8, and in one case Claude Sonnet 4.6, as the grading model in place of OpenAI's (System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026)).

Results

In the paper, scores ran from 16% for GPT-3.5 Turbo and 32% for GPT-4o to 60% for o3, with Grok 3 and Gemini 2.5 Pro ahead of Claude 3.7 Sonnet and Llama 4 Maverick; OpenAI noted that GPT-4.1 nano outperformed GPT-4o at one twenty-fifth of the cost HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025). GPT-5 raised the HealthBench Hard record from o3's 31.6% to 46.2% in August 2025 (GPT-5 System Card (OpenAI, August 2025)). Since then OpenAI's own length-adjusted HealthBench scores have stayed between 50.9 and 58.1 from GPT-5 through GPT-6 Astra, while its HealthBench Professional scores rose from 46.2 to 63.4 (GPT-5.6 Preview System Card (OpenAI, June 2026); GPT-6 Astra System Card (OpenAI, September 2026)). OpenAI also reported HealthBench gains for GPT-5.6 Luna when it opened ChatGPT Health to all U.S. users on July 23, 2026 (Source: techcrunch.com). Other developers report it as a cross-developer comparison point: Anthropic in its system cards from Claude Opus 4.8 onward, and Meta, whose published benchmarks for Muse Spark gave 42.8% on HealthBench Hard (Source: thenextweb.com). After publishing its GPT-6 Astra announcement on September 3, 2026, OpenAI revised two Anthropic models' HealthBench Professional figures upward, Claude Fable 5.1 from 56.6% to 58.1% and Claude Opus 5 from 54.5% to 56.4% (Source: fortune.com).

HealthBench, length-adjusted (OpenAI)

Full 5,000-conversation set, OpenAI's grader and length adjustment; the Setting column gives the unadjusted score and mean reply length in characters.

HealthBench, length-adjusted (Anthropic)

Full set, run by Anthropic with Claude Opus 4.8 as grader, adaptive thinking at max effort, five trials, no tools.

HealthBench

Original unadjusted score on the full set, GPT-4.1 grader unless the Setting says otherwise. Paper rows are means of 16 runs.

HealthBench Hard

Unadjusted score on the 1,000-conversation Hard subset.

HealthBench Hard, length-adjusted (OpenAI)

HealthBench Consensus

Unadjusted score on the Consensus subset.

HealthBench Consensus, length-adjusted (OpenAI)

HealthBench Professional

Length-adjusted score on the 525 clinician tasks. OpenAI rows use OpenAI's grader (GPT-5.4 in the paper); Anthropic rows use a Claude grader, named in the Setting column.

HealthBench Professional — Raw rubric score

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Professional, length-adjusted

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Hard Hallucinations (error rate)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Consensus Urgent (error rate)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Consensus Global Health (error rate)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Consensus (unadjusted)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

HealthBench Professional (unadjusted)

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

A HealthBench score is a share of rubric points, not a percentage of questions answered correctly, and the paper notes that per-conversation criteria are not comprehensive. It is sensitive to reply length. The paper found small within-model correlations between length and score (r = 0.123 for o3) and that o3's win rate over GPT-4.1 fell from 72.9% to 63.7% when only replies of similar length were compared HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025). The HealthBench Professional paper measured a gain of 3.9 points from moving verbosity from low to medium and introduced the length adjustment for that reason HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats (Hicks et al., OpenAI, 2026). The adjustment can reorder models: in Anthropic's September 2026 results Claude Opus 5 has the highest raw HealthBench score (67.1%) and the lowest adjusted score (57.8%) of four models (System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026)). Adjusted and unadjusted figures, and figures produced with different grader models, should not be compared with one another; the same model (Claude Opus 4.8) scores 55.8% and 56.9% on HealthBench Professional under two Claude graders (System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026)).

OpenAI's unadjusted figures for the same model differ between its own system cards. The GPT-5.4 Thinking card gives GPT-5.4 62.6% on HealthBench, 40.1% on HealthBench Hard and 96.6% on HealthBench Consensus, with GPT-5.2 at 63.3%, 42.0% and 94.5% and mean replies of 3,311 and 2,676 characters, and does not name its grader (GPT-5.4 Thinking System Card); the GPT-5.6 preview card gives unadjusted scores of 55.7, 30.3 and 96.4 for GPT-5.4 and 60.7, 38.9 and 94.7 for GPT-5.2 (GPT-5.6 Preview System Card (OpenAI, June 2026)).

The paper also reports reliability. The "worst-at-k" score takes the worst of k sampled replies to each conversation; o3's worst-of-16 score was a third lower than its average, which the authors present as evidence that a model with a good mean can still give poor replies to the same question on some occasions HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

On the original benchmark OpenAI wrote in June 2026 that it believes HealthBench "is approaching a noise ceiling for frontier models" and that improvements on HealthBench Professional had been much more predictive of other held-out evaluations in its internal testing (GPT-5.6 Preview System Card (OpenAI, June 2026)). For HealthBench Professional, the authors caution that difficult and adversarial cases were deliberately over-sampled, so that a moderate score "can coexist with high real-world performance in typical usage" HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats (Hicks et al., OpenAI, 2026).

Limitations and criticism

The evidence gap it does not close

HealthBench measures conversational conduct as judged against physicians' rubrics. It does not measure whether users who receive those conversations experience better health outcomes, and the paper says so: it "does not measure health outcomes of specific workflows," which depend on implementation as well as reply quality, and it calls for real-world studies that measure both HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025). The ChatGPT Health launch page cites no peer-reviewed clinical-outcome studies, a gap Topol/Marcus: LLMs and Patient Outcomes — three-document cluster addresses in finding "very little evidence for LLMs benefiting patients or doctors for health outcomes" outside administrative work (Source: openai.com). A rubric score is evidence about the interaction, not about the clinical endpoint.

Published critique

A viewpoint by Jialin Liu and Siru Liu in Digital Health (October 2025), titled "HealthBench: Advancing AI evaluation in healthcare, but not yet clinically ready," credits the benchmark's scenario coverage, physician involvement and grader validation, and lists four limitations. It relies on synthetic conversations and not real clinical encounters. A model grader may share blind spots with the models it grades, for which the authors recommend human adjudication and graders from different model families. It evaluates static, offline conversations without measuring downstream outcomes. And it omits multimodal data such as laboratory trends and imaging, longitudinal follow-up and the constraints of health-record systems. They argue that a model could score well on emergency-referral items and still misjudge urgency in practice, and propose prospective "silent-mode" trials embedded in electronic health records (Source: pmc.ncbi.nlm.nih.gov).

Limits stated by the authors

The paper acknowledges that physicians often disagreed with one another at every stage, that example-specific criteria were each written by a single physician, that grader validation covered only the consensus criteria, and that the grader prompts were tuned using GPT-4.1, the model chosen as grader. The benchmark, the grader and many of the models scored on it come from the same company HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025).

Use in policy and safety frameworks

OpenAI reports HealthBench in the health sections of its system cards: the GPT-5 system card gives HealthBench, Hard and Consensus results with sub-measures for hallucinations on hard cases, failures in urgent situations and failures to adjust for global-health context; the GPT-5.2 update describes results as similar to GPT-5.1; and the GPT-5.5, GPT-5.6 and GPT-6 Astra cards report it beside dynamic mental-health evaluations. Anthropic reports HealthBench and HealthBench Professional in the capabilities chapters of its system cards from Claude Opus 4.8 onward (System Card: Claude Opus 4.8 (Anthropic, May 2026); System Card: Claude Fable 5 & Claude Mythos 5 (Anthropic, June 2026); System Card: Claude Sonnet 5 (Anthropic, June 2026); System Card: Claude Fable 5.1 & Claude Mythos 5.1 (Anthropic, September 2026)).

The benchmark also bears on how OpenAI positions its consumer health product. OpenAI states that ChatGPT Health is "designed to support, not replace" medical care and is "not intended for diagnosis or treatment," framing that maintains distance from software-as-a-medical-device classification, under which outcome evidence would be required (Source: openai.com). See Healthcare — AI Deployment.

Relationships

• instance-of: AI Benchmarks and Evaluation — a rubric-based alternative to exam-style medical benchmarks.
• related: Healthcare — AI Deployment — the sector context and the ROI-evidence debate.
• related: OpenAI, Topol/Marcus: LLMs and Patient Outcomes — three-document cluster, AI Deskilling.
• related: Muse Spark Safety & Preparedness Report (Meta, May 2026), Muse Spark (Meta Superintelligence Labs) — Meta reported HealthBench Hard among Muse Spark's launch benchmarks.
• related: GPT-5.4 Thinking System Card — reports HealthBench, Hard and Consensus scores with mean reply length for GPT-5.4 and GPT-5.2.
• related: APEX-Agents — Mercor's APEX-1 includes primary-care physician tasks graded by rubric; GDPval — covers nurses and health-services managers among its occupations.

See also

• PaperBench — cited in the paper as an earlier rubric evaluation from OpenAI.
• Virology Capabilities Test, LAB-Bench — biomedical evaluations used for risk assessment, as distinct from care quality.