This is the paper by Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K. Arora and twelve co-authors at OpenAI, with Hicks, Trofimov and Karan Singhal as corresponding authors, that introduces HealthBench Professional, a clinician-focused variant of HealthBench. It was submitted to arXiv as 2604.27470 on 30 April 2026. The benchmark evaluates models on tasks that clinicians brought to OpenAI's ChatGPT for Clinicians product, and reports that the product harness running GPT-5.4 outperformed base GPT-5.4, every other model tested, and physician-written responses.
Summary
Design goals
HealthBench Professional contains 525 physician-authored tasks in three use cases: care consult, writing and documentation, and medical research. Each example asks for the next model response in a single- or multi-turn conversation and is graded against example-specific criteria, as in HealthBench. The authors state three aims: to be meaningful (improvements reflect better performance on real clinician tasks), trustworthy (low scores reflect model limitations rather than evaluation noise) and challenging (substantial headroom for frontier models). They describe it as complementing rather than replacing HealthBench, with narrower coverage but improved data sourcing, review and difficulty sampling.
Data collection
A cohort of 190 physicians with experience across 50 countries and 26 specialties, speaking 52 languages, contributed; 68% were independent-practice or staff physicians, 7% fellows, 19% residents in their third year or above and 5% first- or second-year residents. Conversations came from physicians testing ChatGPT for Clinicians during its development in two modes: good faith (routine clinical, academic, administrative or research use) and red teaming (deliberate adversarial probing using emotionally charged framing, role-play, false premises, obscured intent, format-shifting requests, and questionable diagnoses presented as fact). Red teaming was performed primarily against GPT-5.4 and also GPT-5.2 Instant.
Examples passed through three stages: physician authorship (conversation, initial rubric and a Likert 1–7 rating of the model's final response across accuracy, completeness, clarity, instruction following and overall impact); physician review, in which difficult examples (Likert 1–2) had two physicians independently confirm that the model response contained meaningful flaws and the scenario was realistic; and final adjudication, which removed subjective or preference-based criteria. Every example and criterion was thus reviewed by three or more physicians.
From 15,079 candidate examples, stratified sampling by difficulty produced the final 525, up-weighting difficult examples for OpenAI's frontier models by roughly 3.5 times relative to the underlying distribution (about 8 times for good-faith tasks). About one-third of the benchmark comes from red teaming. The authors note that this "partially confounds comparisons across models" because the selection targeted GPT-5.4. The final set is organised into three slices: good faith typical (Likert 3–7), good faith difficult (Likert 1–2) and red teaming difficult (Likert 1–2). Twenty-eight specialties are represented; cardiology and neurology each account for 8.2% of examples and the top ten specialties for 57.7%.
For a human baseline, a physician with relevant specialty familiarity wrote the best possible next response for every example, with internet and non-AI reference tools, no time limit and no AI assistance, and never for an example they had authored. The authors say these physicians' performance may exceed that of an average physician in the specialty.
Scoring
Criteria carry point values from −10 to 10 and are graded independently by GPT-5.4 at low reasoning effort. Because longer answers have more opportunities to satisfy positive criteria, the primary reported score is length-adjusted: an empirical slope of 2.94 × 10⁻⁵ per character (about 0.0147 per 500 characters) fitted over OpenAI models at several verbosity settings up to 4,000 characters, applied as a penalty for final answers longer than 2,000 characters and a credit for shorter ones. Scores are reported multiplied by 100. Models were run at their highest available reasoning effort with default verbosity and system messages, with 8 samples per example for the main comparisons; paired t-tests with Holm adjustment were used for comparisons.
Results
GPT-5.4 in ChatGPT for Clinicians scored 59.0 overall, ahead of physician-written responses at 43.7 (p = 3.7 × 10⁻¹⁰) and of every other system, the closest being base GPT-5.4 at 48.1 (p = 1.3 × 10⁻⁷). By use case it beat physicians in care consult (51.0 vs 42.7), writing and documentation (64.1 vs 32.1) and medical research (67.0 vs 56.3); against base GPT-5.4 it was statistically similar in care consult (p = 0.72), stronger in writing and documentation (next-best GPT-5.4 at 34.6) and in research (67.0 vs 58.1). On the red-teaming difficult slice it scored 55.8 against 30.0 for physicians and 26.2 for base GPT-5.4; on the good-faith typical slice it scored 69.0 against 55.7 for physicians and 61.4 for Gemini 3.1 Pro, but was statistically indistinguishable from GPT-5, GPT-5.4, GPT-5.2, GPT-5.4 with browsing and Claude Opus 4.7 after Holm correction; on the good-faith difficult slice it was tied with physicians and the field. A simple GPT-5.4-plus-browsing harness scored 45.8 overall. By specialty, the product harness had the highest mean in 6 of 10 plotted specialties, with significant gains over physicians in nephrology (72.1 vs 43.6) and cardiology (53.9 vs 30.7).
Moving verbosity from low to medium added about 1,300 characters and 3.9 unadjusted points on average; after adjustment the comparison was nearly flat. Raising reasoning effort from lowest to highest improved scores by 3.3 points on average across GPT-5 to GPT-5.4, and by 5.6 to 7.3 points for GPT-5.4, which the authors present as test-time compute producing better rather than merely longer answers.
Interpretation and access
The discussion states that because difficult and adversarial cases were deliberately over-sampled, "absolute benchmark scores are not directly comparable to real-world performance rates," and that a moderate aggregate score can coexist with high real-world performance in typical usage. EHR-integrated workflows and institution-specific constraints are not covered, and institutions are advised to run their own pre-deployment evaluations. Results come from an internal implementation; no official external implementation is released, though the simple-evals HealthBench code gained an option matching the internal grader settings.
Key claims
- 525 tasks selected from 15,079 candidates; difficult examples enriched about 3.5×; about one-third red teaming; 190 physicians; every example adjudicated by three or more physicians.
- Grader GPT-5.4 at low reasoning; length adjustment of about 1.47 points per 500 characters relative to a 2,000-character baseline.
- Length-adjusted scores: GPT-5.4 in ChatGPT for Clinicians 59.0; base GPT-5.4 48.1; GPT-5.4 with browsing 45.8; specialty-matched physicians 43.7.
- Scores are not comparable to real-world performance rates because of adversarial sampling.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org); abstract page arxiv.org. Data assets: openaipublic.blob.core.windows.net.
Relationships
- supports: HealthBench — the primary text for the HealthBench Professional variant and its physician baseline.
- depends-on: HealthBench: Evaluating Large Language Models Towards Improved Human Health (Arora et al., OpenAI, 2025) — the rubric-evaluation design and grader validation it builds on.
- instance-of: AI Benchmarks and Evaluation
- related: AI in Healthcare, Healthcare — AI Deployment, OpenAI