Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet (Anthropic, October 2024)

The Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet is a 14-page document Anthropic published on 22 October 2024 as an addendum to the Claude 3 model card, covering two models released that day: an upgraded version of Claude 3.5 Sonnet, referred to throughout as "Claude 3.5 Sonnet (New)", and the new Claude 3.5 Haiku (Source: assets.anthropic.com). Its principal new subject is computer use — the upgraded Sonnet's ability to interpret screenshots of a graphical user interface and generate tool calls to operate it — which the addendum evaluates on OSWorld and treats separately in its safety chapter. The document also introduces SWE-bench Verified, τ-bench, AIME and IFEval to Anthropic's reported evaluation suite, and records that joint pre-deployment testing of the upgraded Sonnet was conducted by the US AI Safety Institute and the UK AI Safety Institute, with an independent assessment by METR.

Models and scope

The addendum describes the upgraded Claude 3.5 Sonnet as improving on its predecessor and, with computer use, setting what Anthropic calls new state-of-the-art results in agentic coding (SWE-bench Verified), agentic task completion (τ-bench) and computer use from screenshots (OSWorld). Claude 3.5 Haiku is presented as performing comparably to the original Claude 3.5 Sonnet and Claude 3 Opus on many tasks, particularly reasoning and instruction following, and launched initially as a text-only model, so no multimodal results are reported for it. The knowledge cutoff is April 2024 for the upgraded Sonnet (unchanged from the original 3.5 Sonnet) and July 2024 for 3.5 Haiku (§1.1). A footnote states that the evaluation tables exclude OpenAI's o1 family because its dependence on extensive pre-response computation makes comparison "outside the scope of this report."

Evaluation approach and findings

Computer use (§2.1). The upgraded Sonnet was evaluated on OSWorld with screenshot-only inputs, omitting the accessibility-tree option the benchmark also offers. Anthropic reports an average success rate of 14.9% at the standard 15-step limit and 22% when the prompt was optimised and the step budget raised to 50, against the benchmark's human success rate of 72.36%; Table 1 breaks both runs down by OSWorld category with 95% confidence intervals. The addendum describes the capability as at an early stage, classified under ASL-2, and states that first-party and independent third-party multimodal red-teaming was conducted.

Tool use and agentic tasks (§2.2). On SWE-bench Verified the upgraded Sonnet scores 49.0% pass@1 and Claude 3.5 Haiku 40.6%, against 33.4% for the original 3.5 Sonnet, 22.2% for Claude 3 Opus and 7.2% for Claude 3 Haiku; a footnote compares the 49.0% with the 45.2% then leading the SWE-bench leaderboard. On τ-bench, which scores pass^k (all k samples correct), the upgraded Sonnet solves 69.2% of retail and 46.0% of airline cases at pass^1, and Claude 3.5 Haiku 51.0% and 22.8%; Figure 1 plots pass^k for retail at k up to 8. On Anthropic's internal agentic coding evaluation, defined in the original Claude 3.5 Sonnet addendum, the two new models solve 78% and 74% of problems.

Vision (§2.3). Table 5 reports the upgraded Sonnet against the original, Claude 3 Opus and Sonnet, GPT-4o and Gemini 1.5 Pro on MMMU (validation), MathVista (testmini), AI2D, ChartQA and DocVQA, all 0-shot, with chain-of-thought on MMMU, MathVista and ChartQA. The addendum describes the results on MathVista, ChartQA, AI2D and MMMU as state of the art; the DocVQA figure for the upgraded Sonnet (94.2%) is below the original's 95.2%.

Refusals (§2.4). Using the Wildchat and XSTest sets from the Claude 3 model card, Table 6 reports correct-refusal rates on Wildchat toxic prompts (89.2% for the upgraded Sonnet, 88.0% for 3.5 Haiku, against 96.4% for the original 3.5 Sonnet) and incorrect-refusal rates on Wildchat non-toxic prompts (5.3% and 4.8%, against 11.0%) and XSTest (4.3% and 2.0%, against 1.7%).

Reasoning, coding and question answering (§2.5). Tables 7 and 8 cover GPQA Diamond, MMLU (5-shot, 5-shot CoT and 0-shot CoT), MMLU Pro, MATH, HumanEval, MGSM, DROP, BIG-Bench Hard, AIME 2024 (0-shot CoT and Maj@64) and IFEval. The upgraded Sonnet is compared with the original, Claude 3 Opus and Sonnet, GPT-4o, Gemini 1.5 Pro (September 2024 figures) and Llama 3.1 405B; Claude 3.5 Haiku with Claude 3 Haiku, GPT-4o mini and Gemini 1.5 Flash. Peer figures are taken from the developers' own publications, as the table footnotes state.

Human feedback (§2.6). Raters compared each model with the original Claude 3.5 Sonnet as baseline across coding, creative writing, documents, multilingual, instruction following, honesty, vision instruction following and harmlessness (Figure 2). The text states that the upgraded Sonnet wins 61% on document analysis, 57% on visual understanding, 58% on creative writing, 52% on coding and 51% on precise instruction following, and that Claude 3.5 Haiku beats Claude 3 Opus "by a significant margin" in coding and document analysis.

Safety

The Trust & Safety chapter (§3.1) reports red-teaming across fourteen policy areas in six languages (English, Arabic, Spanish, Hindi, Tagalog and Chinese). Anthropic states that the upgraded Sonnet's overall harm rates are similar to, and slightly improved over, the original 3.5 Sonnet's, that 3.5 Haiku improved on Claude 3 Haiku particularly for non-English prompts, and that both models "struggled with nuanced requests or those framed as fiction, roleplaying, or artistic content." No numeric harm rates are given. Both models were trained on internal prompt-injection test sets (§3.1.2), and the addendum recommends a dedicated virtual machine, limited data access, restricted internet domains and a human in the loop for computer use. Computer-use red-teaming (§3.1.3) identified scaled account creation, scaled content distribution, age-assurance bypass and abusive form filling as potential abuse vectors, none deemed imminent.

Section 3.1.4 records Anthropic's reasoning that computer use did not change its RSP threat models: the capability alone was judged unlikely to raise CBRN risk without knowledge and skills performance, unlikely to add cyber capability beyond existing tools while possibly lowering the barrier for novices, and not on the critical path to the software-engineering skills tested as autonomy precursors. The frontier-risk chapter (§3.2) describes CBRN knowledge tests and uplift assessment with new baselines and manual red-teaming, capture-the-flag challenges across pwn, reverse engineering, cryptography, web and network security, and software-engineering tasks as an autonomy precursor, using "improved threat models, new evaluation techniques, and stronger elicitation methods" than earlier releases. The stated result (§3.2.1) is that the upgraded Sonnet "showed increased capabilities in risk-relevant areas" across all three domains but "did not demonstrate capabilities requiring ASL-3 safeguards," and that the models did not trigger the full RSP evaluation protocol (§3.3). No scores from the frontier-risk evaluations, the AISI testing or the METR assessment are reported.

Provenance and confidence

The raw file is the text layer of the PDF extracted with pdftotext -layout; table columns survive, but Figure 1 (τ-bench pass^k) and the per-model bars of Figure 2 are not reproduced in a form that attaches values to models. All results are Anthropic's own, with peer figures quoted from OpenAI, Google and Meta publications. Confidence is medium: a single developer-authored source whose benchmark claims are of a generation since superseded.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

Relationships

• supports: Claude 3.5 Haiku (primary reference for its launch evaluations), Claude 3 Opus (comparison rows)
• depends-on: The Claude 3 Model Family: Opus, Sonnet, Haiku (Anthropic model card, March 2024) (the addendum extends it; the refusal and human-feedback methods are defined there)
• related: OSWorld, SWE-bench, τ-bench, GPQA, MMLU, AIME (as a model benchmark), GPT-4 Family (OpenAI), Llama 3 (Meta)
• instance-of: AI System Cards