This paper introduces the Virology Capabilities Test, a benchmark measuring whether a language model can troubleshoot complex virology laboratory protocols. Its primary authors are Jasper Götting, Pedro Medeiros and Jon G Sanders of SecureBio, with Seth Donoughe of SecureBio as corresponding author; Nathaniel Li, Long Phan and Dan Hendrycks contributed from the Center for AI Safety, Karam Elabd from SecureBio and Lennart Justen from the MIT Media Lab. It was posted to arXiv as 2504.16137 on 22 April 2025.
The authors' stated target is knowledge that standard benchmarks miss. They argue that four-option multiple-choice benchmarks such as MMLU and GPQA "do not capture rare, tacit, and search-proof knowledge," do not test reasoning over images, and suffer from label errors and saturation, while existing biosecurity assessments rest on a few benchmarks and costly human red-teaming exercises. VCT covers virology methods with dual-use potential and closely related methods; it excludes general molecular and cellular biology, and on safety grounds excludes material the authors judged unambiguously hazardous.
Summary
Question creation
Of 184 experts who registered interest, 105 attended a training webinar, 86 were fully onboarded and 68 began contributing questions; each was vetted on their first three submissions, leaving 57 contributing virologists. A custom Airtable database managed submission, review and baselining, with a review structure and payout schema the authors say were inspired by GPQA. Every submitted question was reviewed by two other experts familiar with its topic, who approved or disapproved it and gave feedback; of 1,042 reviews, 72.6% approved and 27.4% disapproved. Contribution was heavy-tailed: 41 experts submitted five or fewer questions, and the ten most productive contributed 51% of the total.
Of 507 submissions (383 image-based, 124 text-only), 45 were disapproved twice, 54 were abandoned mid-process, and editors removed a further 14 that passed review but did not meet quality criteria. Each of the 394 remaining questions was then answered by three or four non-experts, graduate students and post-graduate researchers with a STEM degree outside biology, and 29 questions answered correctly by more than two-thirds of them were excluded as web-searchable. The remaining 365 were split into the 322-question shared set and a held-out set.
Contributors wrote several structured components per question so that the same input can be run in more than one format. The recommended format is multiple-response: the answerer receives the question text, an optional image, and between four and ten true-or-false statements, and must select exactly the true ones. The authors note the guessing probability for n statements is 1/(2^n − 1), far below a four-option multiple-choice question, and grading is all-or-nothing. Alongside accuracy they report a mismatch count, the number of statements on which a response differs from the correct set.
Evaluation and results
Models were evaluated zero-shot. Anthropic, Google and non-reasoning OpenAI models were run through the UK AI Safety Institute's Inspect framework with its built-in multiple-choice solver and scorer; other models used custom scripts. Reported accuracies are means of three independent runs, with standard errors below 1% for all models.
For the human baseline, 36 of the 57 experts were available after question submission closed. They answered only questions they had not seen and for which they had declared expert-level familiarity, with 15 to 30 minutes per question and any resources except LLMs or other people; 229 questions were answered by three experts, 65 by two, 9 by one and 19 by none.
Table 1 reports o3 at 43.8% accuracy (mismatch count 1.06, 94th percentile against the expert pool), Gemini 2.5 Pro (experimental) 37.6%, o4-mini 37.0%, o1 35.4%, Claude 3.5 Sonnet (October 2024) 33.6%, Claude 3.7 Sonnet 30.8% (up to 4,096 reasoning tokens), GPT-4.5 Preview 28.3%, Gemini 2.0 Pro (experimental) 27.4%, GPT-4.1 26.6%, Claude 3.5 Sonnet (June 2024) 26.9%, Gemini 1.5 Pro 25.4% and GPT-4o (November 2024) 18.8%. The expert virologists averaged 22.1% with a mismatch count of 1.71, below every model tested except GPT-4o.
Because each expert answered a set tailored to their own specialties, the authors also evaluated models on each expert's individual question set, for the 36 experts who answered at least 10 questions. All frontier models scored above the median expert, and o3 outperformed 34 of 36. The authors suggest experts would have scored lower still on randomly assigned questions, so the comparison may understate the gap.
Discussion
The authors write that the excluded dual-use topics share "similar essential features" with those included, so VCT can act as a proxy measure of potentially hazardous knowledge during pre-deployment testing. They state a limitation directly: "Our benchmark, by itself, does not directly assess the capabilities of humans who draw upon that model for assistance in real-world virology work," and note that turning realistic scenarios into well-formed questions may lose some relevance, and that embodied and communal tacit knowledge are not represented.
Their policy position is access control rather than capability limits: their "tentative view is that an AI's ability to provide expert-level troubleshooting on highly dual-use methods should itself be considered a highly dual-use technology." They judge a text-only expert-level virology assistant to pose less risk than an autonomous agent performing tasks, while arguing both warrant controls, and propose that such expertise be excluded from widely available next-generation models while remaining accessible through know-your-customer mechanisms to researchers at institutions with appropriate safety protocols. They name the National Science Advisory Board for Biosecurity as a body positioned to address dual-use biology skills in AI systems, and suggest VCT and similar evaluations could serve AI safety institutes and non-governmental organisations adapting dual-use frameworks.
Key claims
- 322 multimodal questions in the shared set, from 507 submissions by 57 vetted PhD virologists, after two-stage peer review, editing and non-expert screening.
- Recommended format is multiple-response with 4–10 true/false statements, all-or-nothing grading; guessing probability 1/(2^n − 1).
- Expert baseline 22.1% within experts' own specialties, with internet and literature access and up to 30 minutes per question.
- Best model at publication: o3 at 43.8%, in the 94th percentile of 36 experts; all frontier models scored above the median expert.
- The full question set is not publicly released.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org.
Relationships
- supports: Virology Capabilities Test — the primary text behind the benchmark's description and its original results.
- instance-of: AI Benchmarks and Evaluation
- depends-on: GPQA: A Graduate-Level Google-Proof Q&A Benchmark — the review and payout structure the authors say inspired theirs.
- related: LAB-Bench, WMDP — adjacent benchmarks of biology-related model capability.
- related: Center for AI Safety (CAIS), UK AI Safety Institute (AI Security Institute)