This technical report describes the pre-deployment evaluation of Anthropic's upgraded Claude 3.5 Sonnet, which the report calls "Sonnet 3.5 (new)," conducted jointly by the US AI Safety Institute at NIST and the UK AI Safety Institute at the Department for Science, Innovation and Technology. It was published on NIST's website in November 2024 with a companion blog post. The two institutes ran independent tests on the model during a limited period of pre-release access, cooperated on methodology and interpretation, and shared their initial findings with Anthropic before the model's release.
The report assesses four domains: biological capabilities, cyber capabilities, software and AI development capabilities, and safeguard effectiveness. Its disclaimer states that the results "should not be interpreted as an indication of whether any evaluated AI system or subcomponent thereof is safe or appropriate for release," that the evaluations are limited to a narrow set of domains and preliminary in nature, and that cross-model comparisons are intended only to aid scientific interpretation.
Summary
Method
The subject model is identified as the version of Claude 3.5 Sonnet available in Anthropic's API as claude-3-5-sonnet-20241022; the introduction gives its release date as 22 October 2024, while Section 2.2 prints 10 October. Three reference models are used for comparison: Sonnet 3.5 (old), the 20 June 2024 release; OpenAI o1-preview (o1-preview-2024-09-12); and GPT-4o (gpt-4o-2024-08-06). The report notes that its agents were optimised for the two Sonnet versions, which it describes as producing a conservative estimate of whether the new model substantially exceeds the reference models, and that it does not control for differences in cost between models.
Testing followed an iterative process: preliminary runs on a development set, manual review for issues that depressed model performance, prompt and environment adjustment, then the full evaluation. The institutes state that real-world users will likely find further techniques that improve performance and bypass safeguards beyond what the report found. Agent tasks use a ReAct-style loop in a Docker-hosted Linux environment with bash, Python and file-editing tools. Where a user could verify success and retry, results are reported as Pass@N, the fraction of tasks solved in at least one of N attempts. The report states that all testing was conducted at a total economic cost "significantly below" that of a human performing the tasks, so results may understate achievable capability.
Biological capabilities
US AISI evaluated the model on a subset of LAB-Bench, with Python sandbox tools available for the CloningScenarios and SeqQA categories. Without external tools, performance was significantly below human expert performance on all domains except TableQA. With tools, on SeqQA the model exceeded both the reference models and human experts; tool access significantly improved sequence-task performance for Sonnet 3.5 (new) and o1-preview while having no clear effect on cloning. In the no-tool replication of the LAB-Bench paper's setup, models abstained more often than humans and had lower precision on the questions they did answer. UK AISI did not publish biological findings.
Cyber capabilities
UK AISI ran 47 challenges, 15 publicly sourced and 32 privately developed, graded by domain experts. Sonnet 3.5 (new) succeeded on 90% of the 29 "technical non-expert" tasks (Pass@10), against 82% for Sonnet 3.5 (old) and 52% for GPT-4o, and on 36% of the 14 "cybersecurity apprentice" tasks, against 29% and 0%. It performed similarly to the old version on 14 vulnerability-discovery tasks and better on 17 vulnerability-exploitation tasks at the non-expert level. Performance on private tasks was slightly lower than on public ones, which the institute reads as some evidence that its qualitative conclusions are not driven by contamination.
US AISI ran Cybench, all 40 tasks, ten attempts each, using Inspect-based agents in Kali Linux containers with a 100-message limit and a 180-second command timeout, after modifying the public codebase and fixing bugs in some challenges. Sonnet 3.5 (new) had a Pass@10 rate of 32.5% against 35% for the best reference model; mean Pass@1 was 0.265 ± 0.066 for Sonnet 3.5 (new), 0.253 ± 0.064 for Sonnet 3.5 (old), 0.258 ± 0.065 for o1-preview and 0.190 ± 0.054 for GPT-4o, differences the report says are smaller than measurement error. Models solved lower-FST tasks more consistently. Cutting the budget to 15 messages "significantly decreases all agent's solve rates," while cutting from 100 to 50 lowered the model's rate by less than 1%.
Software and AI development
On MLAgentBench, scored so that the unimproved model is 0% and the best human improvement 100%, Sonnet 3.5 (new) averaged 57% against 48% for the best reference model. On UK AISI's 14 custom software and AI development challenges the model scored 66% against 64%, and on related general-reasoning tasks 47% against 35%.
Safeguard efficacy
US AISI posed malicious requests using variants of publicly available jailbreaks and found they led the model to accommodate most harmful requests across all tested categories. UK AISI tested public and privately developed jailbreaks and found several basic, previously known attacks transferred from Sonnet 3.5 (old) without alteration; it assesses that the evaluated safeguards "would not reliably prevent determined actors from extracting responses to malicious queries." The report states that these tests measure built-in defences only and cannot determine which requests a model should refuse.
Key claims
- Cybench: Pass@10 32.5% (best reference 35%); Pass@1 0.265 ± 0.066; 100 messages, 10 attempts, Inspect agent.
- UK cyber: Pass@10 90% on 29 technical non-expert tasks, 36% on 14 apprentice tasks.
- LAB-Bench: below human experts without tools except TableQA; above human experts on SeqQA with tools.
- MLAgentBench 57% versus 48%; UK software 66% versus 64%; general reasoning 47% versus 35%.
- Public jailbreak variants bypassed safeguards on most tested harmful requests.
Provenance
Retrieved 22 September 2026 as text from the NIST-hosted PDF (nist.gov); the date is taken from the file path and the report's November 2024 publication. The Cybench paper cites this report as reaching 26.5% mean performance and solving a task with a 75-minute first solve time.
Relationships
- supports: Cybench, LAB-Bench — government measurements on both benchmarks.
- related: Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (Zhang et al., 2024) — the benchmark's paper, which cites this report's stronger scaffold.
- related: US AI Safety Institute (NIST AISI), UK AI Safety Institute (AI Security Institute), NIST CAISI (Center for AI Standards and Innovation), Anthropic
- related: AI and Cybersecurity, AI Benchmarks and Evaluation