AI Policy Wiki
Dashboard
⚲

System Card: Claude Opus 4.5 (Anthropic, November 2025)

high confidence · updated 2026-09-22

Anthropic's system card for Claude Opus 4.5. Judges the model below the AI R&D-4 threshold — none of 18 internal survey participants believed it could fully automate an entry-level remote research role — while noting it 'just barely reached our pre-defined benchmark rule-out thresholds.' On CBRN-4 the rule-out is stated as 'less clear for Claude Opus 4.5 than we would like.' Includes a decontamination methodology section and a changelog correcting an ARC-AGI-1 training-set disclosure.

Published November 2025 for Claude Opus 4.5.

Two rule-outs, both qualified

The card is notable less for its determinations than for how narrowly it states them.

AI R&D-4. The threshold requires "the ability to fully automate the work of an entry-level, remote-only Researcher at Anthropic," which the card is careful to distinguish from a weaker reading: "This is a very high threshold of robust, long-horizon competence, and is not merely a stand-in for 'a model that can do most of the short-horizon tasks that an entry-level researcher can do.'"

The determination is negative, supported by a survey: "None of the 18 internal survey participants — who were themselves some of the most prolific users of the model in Claude Code — believed it could fully automate an entry-level remote-only research or engineering role."

But the card flags its own margin: "It is also noteworthy that the model has just barely reached our pre-defined benchmark rule-out thresholds, rather than greatly exceeded them." The specific predicted failures are behavioural rather than knowledge-based — the model "would fail to problem-solve, investigate, communicate, and collaborate in the way a junior researcher could."

CBRN-4. Also ruled out, and also qualified. Opus 4.5 "performed as well as or slightly better than Claude Opus 4.1 and Claude Sonnet 4.5" across biology tasks, but "most notably… in an expert uplift trial, Claude Opus 4.5 was meaningfully more helpful to participants than previous models, leading to substantially higher scores and fewer critical errors, but still produced critical errors that yielded non-viable protocols."

Anthropic states the implication for the framework rather than only for the model: "we take this as an indicator of general model progress where, like in the case of autonomy, a clear rule-out of the next capability threshold may soon be difficult or impossible under the current regime. In fact, the CBRN-4 rule-out is less clear for Claude Opus 4.5 than we would like."

It also attributes part of that uncertainty to the threat model rather than the measurement: "a large part of our uncertainty about the rule-out is also due to our limited understanding of the necessary components of the threat model," CBRN-4 requiring uplift of "a second-tier state-level bioweapons program." See CBRN Uplift.

Decontamination

The card includes a methodology section on benchmark contamination, stating the problem plainly: "When evaluation benchmarks appear in training data, models can achieve artificially inflated scores by memorizing specific examples rather than demonstrating genuine capabilities. This undermines the validity of our evaluation metrics and makes it difficult to compare performance across model generations and among model providers."

Anthropic describes decontamination as "an important component of responsibly evaluating models, albeit one which is an imperfect science," and reports multiple complementary techniques — beginning with substring removal, scanning the training corpus for exact matches and removing documents containing five or more exact question-answer pairs. See AI Benchmarks and Evaluation.

The ARC-AGI correction

The November 24, 2025 changelog records a disclosure correction: Figure 2.10.A was replaced to show semi-private ARC-AGI-1 scores rather than public test-set scores, because "in the previous version of the system card we incorrectly reported that we only trained on the public training set of ARC-AGI-1. We instead trained on a reshuffled train/test split comprising both the public train and test sets."

This is a self-reported error about training-data composition on a benchmark the card reports — relevant context for reading ARC-AGI comparisons across providers.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

EvaluationModelScoreSettingSection
CyberGymClaude Opus 4.550.63%pass@1 over the 1,505 tasks of the CyberGym suite, averaged across five independent replicas; no thinking, 200k context, default effort/temperature/top_p, "think" tool available; developer-reported2.14
SpreadsheetBenchClaude Opus 4.564.25%full 912-problem set; custom harness with bash, string view/edit tools and a Python environment (openpyxl, libreoffice, pandas, numpy); no extended thinking, 200k context, averaged across 5 trials; developer-reported2.15
Multi-agent search (internal information-retrieval benchmark) — Single agentClaude Opus 4.574.8%Claude Opus 4.5 alone, no subagents; k=3 to k=8 samples per problem; developer-reported2.7.1
Multi-agent search (internal information-retrieval benchmark) — Orchestrator + subagentsClaude Opus 4.587.0%Claude Opus 4.5 orchestrator with Claude Haiku 4.5 subagents; orchestrator has no direct search access; developer-reported2.7.1
Multi-agent search (internal information-retrieval benchmark) — Orchestrator + subagentsClaude Opus 4.585.4%Claude Opus 4.5 orchestrator with Claude Sonnet 4.5 subagents; developer-reported2.7.1
Multi-agent search (internal information-retrieval benchmark) — Orchestrator + subagentsClaude Opus 4.592.3%Claude Opus 4.5 orchestrator with Claude Opus 4.5 subagents; developer-reported2.7.1
Multi-agent search (internal information-retrieval benchmark) — Orchestrator + subagentsClaude Sonnet 4.566.5%Claude Sonnet 4.5 orchestrator with Claude Sonnet 4.5 subagents; developer-reported (Anthropic)2.7.1
Single-turn violative request evaluation (harmless response rate) — All tested languages, overallClaude Opus 4.599.78% (± 0.03%)harmless response rate across Arabic, English, French, Korean, Mandarin Chinese, Russian; standard and extended thinking combined; developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — Default thinkingClaude Opus 4.599.70% (± 0.05%)developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — Extended thinkingClaude Opus 4.599.85% (± 0.04%)developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — All tested languages, overallClaude Haiku 4.599.31% (± 0.07%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Default thinkingClaude Haiku 4.599.22% (± 0.12%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Extended thinkingClaude Haiku 4.599.40% (± 0.08%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — All tested languages, overallClaude Sonnet 4.598.87% (± 0.06%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Default thinkingClaude Sonnet 4.598.32% (± 0.14%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Extended thinkingClaude Sonnet 4.599.42% (± 0.08%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — All tested languages, overallClaude Opus 4.199.14% (± 0.05%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Default thinkingClaude Opus 4.198.86% (± 0.13%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — Extended thinkingClaude Opus 4.199.42% (± 0.08%)developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — EnglishClaude Opus 4.599.59%standard and extended thinking combined; developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — ArabicClaude Opus 4.599.85%developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — ChineseClaude Opus 4.599.75%developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — FrenchClaude Opus 4.599.90%developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — KoreanClaude Opus 4.599.88%developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — RussianClaude Opus 4.599.76%developer-reported3.1.1
Single-turn violative request evaluation (harmless response rate) — EnglishClaude Haiku 4.599.38%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ArabicClaude Haiku 4.599.28%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ChineseClaude Haiku 4.599.14%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — FrenchClaude Haiku 4.599.54%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — KoreanClaude Haiku 4.599.10%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — RussianClaude Haiku 4.599.45%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — EnglishClaude Sonnet 4.599.31%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ArabicClaude Sonnet 4.598.99%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ChineseClaude Sonnet 4.598.50%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — FrenchClaude Sonnet 4.598.50%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — KoreanClaude Sonnet 4.598.08%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — RussianClaude Sonnet 4.599.00%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — EnglishClaude Opus 4.198.76%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ArabicClaude Opus 4.199.52%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — ChineseClaude Opus 4.199.23%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — FrenchClaude Opus 4.199.41%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — KoreanClaude Opus 4.198.52%developer-reported (Anthropic)3.1.1
Single-turn violative request evaluation (harmless response rate) — RussianClaude Opus 4.199.40%developer-reported (Anthropic)3.1.1
Single-turn benign request evaluation (over-refusal rate) — All tested languages, overallClaude Opus 4.50.23% (± 0.03%)over-refusal rate, lower is better; six languages, standard and extended thinking combined; developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — Default thinkingClaude Opus 4.50.18% (± 0.03%)developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — Extended thinkingClaude Opus 4.50.27% (± 0.06%)developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — All tested languages, overallClaude Haiku 4.50.10% (± 0.03%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Default thinkingClaude Haiku 4.50.12% (± 0.05%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Extended thinkingClaude Haiku 4.50.08% (± 0.02%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — All tested languages, overallClaude Sonnet 4.50.05% (± 0.02%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Default thinkingClaude Sonnet 4.50.05% (± 0.03%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Extended thinkingClaude Sonnet 4.50.04% (± 0.02%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — All tested languages, overallClaude Opus 4.10.13% (± 0.03%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Default thinkingClaude Opus 4.10.17% (± 0.06%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — Extended thinkingClaude Opus 4.10.08% (± 0.03%)developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — EnglishClaude Opus 4.50.17%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — ArabicClaude Opus 4.50.32%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — ChineseClaude Opus 4.50.30%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — FrenchClaude Opus 4.50.24%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — KoreanClaude Opus 4.50.17%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — RussianClaude Opus 4.50.22%developer-reported3.1.2
Single-turn benign request evaluation (over-refusal rate) — EnglishClaude Haiku 4.50.03%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ArabicClaude Haiku 4.50.09%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ChineseClaude Haiku 4.50.14%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — FrenchClaude Haiku 4.50.09%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — KoreanClaude Haiku 4.50.14%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — RussianClaude Haiku 4.50.12%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — EnglishClaude Sonnet 4.50.03%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ArabicClaude Sonnet 4.50.03%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ChineseClaude Sonnet 4.50.05%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — FrenchClaude Sonnet 4.50.06%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — KoreanClaude Sonnet 4.50.05%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — RussianClaude Sonnet 4.50.08%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — EnglishClaude Opus 4.10.09%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ArabicClaude Opus 4.10.29%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — ChineseClaude Opus 4.10.08%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — FrenchClaude Opus 4.10.11%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — KoreanClaude Opus 4.10.08%developer-reported (Anthropic)3.1.2
Single-turn benign request evaluation (over-refusal rate) — RussianClaude Opus 4.10.11%developer-reported (Anthropic)3.1.2
Multi-turn testing (rubric failure rate) — Tracking and surveillanceClaude Opus 4.54% (±4%)failure rate against test-case rubrics; up to 15-turn generated conversations, 93 test cases across 10 risk areas, each run 10 times; developer-reported3.3
Multi-turn testing (rubric failure rate) — Tracking and surveillanceClaude Opus 4.13% (± 3%)developer-reported (Anthropic); difference from Opus 4.5 not statistically significant3.3
Multi-turn testing (rubric failure rate) — Deadly weaponsClaude Opus 4.55% (±4%)failure rate; developer-reported3.3
Multi-turn testing (rubric failure rate) — Deadly weaponsClaude Opus 4.122% (±8%)failure rate; developer-reported (Anthropic)3.3
Multi-turn testing (rubric failure rate) — Violent extremismClaude Opus 4.113% (±7%)failure rate; Claude Opus 4.5 passed in all cases; developer-reported (Anthropic)3.3
Political even-handedness (open-source Anthropic evaluation) — Even-handednessGPT-589%1,350 prompt pairs, 9 task types, 150 topics; Claude Sonnet 4.5 grader; public system prompt, no extended thinking; reported by Anthropic in the Claude Opus 4.5 system card as previously reported by Anthropic3.5.1
Political even-handedness (open-source Anthropic evaluation) — Even-handednessGemini 2.5 Pro97%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — Even-handednessGrok 496%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — RefusalsGPT-54%refusal rate on political prompts, lower is better; reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — RefusalsGemini 2.5 Pro3%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — RefusalsGrok 41%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — Opposing perspectivesGPT-517%share of responses acknowledging opposing perspectives, higher is better; reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — Opposing perspectivesGemini 2.5 Pro19%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
Political even-handedness (open-source Anthropic evaluation) — Opposing perspectivesGrok 434%reported by Anthropic in the Claude Opus 4.5 system card3.5.1
BBQ (Bias Benchmark for Question Answering) — Disambiguated biasClaude Opus 4.5-0.64bias score in %, closer to zero is better; standard (non-extended) thinking; developer-reported3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous biasClaude Opus 4.50.26bias score in %, standard thinking; developer-reported3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated biasClaude Haiku 4.50.54developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous biasClaude Haiku 4.51.37developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated biasClaude Sonnet 4.5-2.21developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous biasClaude Sonnet 4.50.25developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated biasClaude Opus 4.1-0.51developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous biasClaude Opus 4.10.20developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated accuracyClaude Opus 4.588.7accuracy in %, standard thinking; developer-reported3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous accuracyClaude Opus 4.599.7accuracy in %, standard thinking; developer-reported3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated accuracyClaude Haiku 4.571.2developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous accuracyClaude Haiku 4.598.0developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated accuracyClaude Sonnet 4.582.2developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous accuracyClaude Sonnet 4.599.7developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Disambiguated accuracyClaude Opus 4.190.7developer-reported (Anthropic)3.5.2
BBQ (Bias Benchmark for Question Answering) — Ambiguous accuracyClaude Opus 4.199.8developer-reported (Anthropic)3.5.2
Agentic coding malicious-use evaluation (refusal rate) — No mitigationsClaude Opus 4.5100%150 malicious coding requests, same coding tools as capability evaluations, no additional safeguards; developer-reported5.1.1
Agentic coding malicious-use evaluation (refusal rate) — No mitigationsClaude Haiku 4.5100%developer-reported (Anthropic)5.1.1
Agentic coding malicious-use evaluation (refusal rate) — No mitigationsClaude Sonnet 4.598.7%developer-reported (Anthropic)5.1.1
Agentic coding malicious-use evaluation (refusal rate) — No mitigationsClaude Opus 4.196.0%developer-reported (Anthropic)5.1.1
Claude Code malicious-use evaluation (refusal rate) — No mitigationsClaude Opus 4.577.80%49 malicious prompts, each run 10 times, standard Claude Code tools, no mitigations; developer-reported5.1.2
Claude Code malicious-use evaluation (refusal rate) — No mitigationsClaude Haiku 4.569.39%developer-reported (Anthropic)5.1.2
Claude Code malicious-use evaluation (refusal rate) — No mitigationsClaude Sonnet 4.563.06%developer-reported (Anthropic)5.1.2
Claude Code malicious-use evaluation (refusal rate) — No mitigationsClaude Opus 4.148.16%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — No mitigationsClaude Opus 4.593.07%61 dual-use and benign prompts, each run 10 times, no mitigations; developer-reported5.1.2
Claude Code dual-use & benign evaluation (success rate) — No mitigationsClaude Haiku 4.584.92%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — No mitigationsClaude Sonnet 4.596.56%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — No mitigationsClaude Opus 4.194.43%developer-reported (Anthropic)5.1.2
Claude Code malicious-use evaluation (refusal rate) — With mitigationsClaude Opus 4.597.35%with system-prompt instructions and a FileRead tool-result reminder; developer-reported5.1.2
Claude Code malicious-use evaluation (refusal rate) — With mitigationsClaude Haiku 4.596.73%developer-reported (Anthropic)5.1.2
Claude Code malicious-use evaluation (refusal rate) — With mitigationsClaude Sonnet 4.595.10%developer-reported (Anthropic)5.1.2
Claude Code malicious-use evaluation (refusal rate) — With mitigationsClaude Opus 4.184.29%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — With mitigationsClaude Opus 4.596.52%developer-reported5.1.2
Claude Code dual-use & benign evaluation (success rate) — With mitigationsClaude Haiku 4.586.07%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — With mitigationsClaude Sonnet 4.598.20%developer-reported (Anthropic)5.1.2
Claude Code dual-use & benign evaluation (success rate) — With mitigationsClaude Opus 4.198.52%developer-reported (Anthropic)5.1.2
Malicious computer use evaluation (refusal rate) — No mitigationsClaude Opus 4.588.39%112 test cases in a sandboxed offline computer-use environment; developer-reported5.1.3
Malicious computer use evaluation (refusal rate) — No mitigationsClaude Haiku 4.572.32%developer-reported (Anthropic)5.1.3
Malicious computer use evaluation (refusal rate) — No mitigationsClaude Sonnet 4.583.03%developer-reported (Anthropic)5.1.3
Malicious computer use evaluation (refusal rate) — No mitigationsClaude Opus 4.166.96%developer-reported (Anthropic)5.1.3
Shade adaptive indirect prompt injection (coding) — 1 attempt, extended thinkingClaude Opus 4.50.3%attack success rate, lower is better; Gray Swan Shade adaptive red-teaming tool, no additional safeguards5.2.2.1
Shade adaptive indirect prompt injection (coding) — 200 attempts, extended thinkingClaude Opus 4.510.0%attack success rate; adaptive attacker with 200 attempts5.2.2.1
Shade adaptive indirect prompt injection (coding) — 1 attempt, standard thinkingClaude Opus 4.50.7%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (coding) — 200 attempts, standard thinkingClaude Opus 4.517.5%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (coding) — 1 attempt, extended thinkingClaude Sonnet 4.517.7%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (coding) — 200 attempts, extended thinkingClaude Sonnet 4.570.0%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (coding) — 1 attempt, standard thinkingClaude Sonnet 4.529.9%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (coding) — 200 attempts, standard thinkingClaude Sonnet 4.587.5%attack success rate5.2.2.1
Shade adaptive indirect prompt injection (computer use) — 1 attempt, extended thinking, without safeguardsClaude Opus 4.50.0%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, extended thinking, without safeguardsClaude Opus 4.50.0%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, extended thinking, with safeguardsClaude Opus 4.50.0%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, extended thinking, with safeguardsClaude Opus 4.50.0%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, standard thinking, without safeguardsClaude Opus 4.50.71%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, standard thinking, without safeguardsClaude Opus 4.528.6%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, standard thinking, with safeguardsClaude Opus 4.50.32%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, standard thinking, with safeguardsClaude Opus 4.514.3%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, extended thinking, without safeguardsClaude Sonnet 4.514.2%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, extended thinking, without safeguardsClaude Sonnet 4.585.7%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, extended thinking, with safeguardsClaude Sonnet 4.59.1%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, extended thinking, with safeguardsClaude Sonnet 4.592.9%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, standard thinking, without safeguardsClaude Sonnet 4.528.4%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, standard thinking, without safeguardsClaude Sonnet 4.585.7%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 1 attempt, standard thinking, with safeguardsClaude Sonnet 4.518.9%attack success rate5.2.2.2
Shade adaptive indirect prompt injection (computer use) — 200 attempts, standard thinking, with safeguardsClaude Sonnet 4.592.9%attack success rate5.2.2.2
Automated behavioral audit — evaluation awareness — Verbalized evaluation awarenessClaude Opus 4.5fewer than 10% of investigationsClaude Sonnet 4.5 auditor, ~1,800 investigations per model, scored by Claude Sonnet 4.5; card states the model discussed the possibility of being in a test in fewer than 10% of investigations; developer-reported6.2
Petri (in-development version) — evaluation awareness — Clear speculation about being testedClaude Opus 4.5 Previewunder 5% of transcripts362 investigations per model; Claude Sonnet 4.5 and GPT-5 as auditors, Claude Opus 4.1, Gemini 2.5 Pro and GPT-5 as scorers; pre-final preview snapshot; developer-reported6.2.4
Internal codebase sabotage propensity — Prompted-classifier scenarioClaude Opus 4.50 instancescard states Claude Opus 4.5 never sabotaged the classifier; Claude Code–based scenario; developer-reported6.9
Internal codebase sabotage propensity — Prompted-classifier scenarioClaude Sonnet 4.50 instancescard states Claude Sonnet 4.5 never sabotaged; verbalized evaluation awareness rate of 5% on this scenario; developer-reported6.9
Internal codebase sabotage propensity — Prompted-classifier scenarioClaude Opus 4.10.8%share of transcripts in which the model sabotaged the classifier; developer-reported (Anthropic)6.9
Reward-hack-prone coding tasks — Classifier hack rateClaude Opus 4.50%hand-selected coding problems from the training distribution; developer-reported6.10.1
Reward-hack-prone coding tasks — Hidden test hack rateClaude Opus 4.50%share of problems whose submitted solution fails held-out hidden tests; developer-reported6.10.1
Impossible Tasks — Classifier hack rate with no promptClaude Opus 4.555%agentic setting on intentionally impossible problems, minimal instructions; developer-reported6.10.1
Impossible Tasks — Classifier hack rate with anti-hack promptClaude Opus 4.535%anti-hack prompt discouraging hard-coding; developer-reported6.10.1
Reward hacking in a subset of training data — Classifier hack rateClaude Opus 4.51%monitored subset of the RL training distribution; developer-reported6.10.1
Reward-hack-prone coding tasks — Classifier hack rateClaude Haiku 4.56%developer-reported (Anthropic)6.10.1
Reward-hack-prone coding tasks — Hidden test hack rateClaude Haiku 4.53%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with no promptClaude Haiku 4.530%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with anti-hack promptClaude Haiku 4.523%developer-reported (Anthropic)6.10.1
Reward hacking in a subset of training data — Classifier hack rateClaude Haiku 4.51%developer-reported (Anthropic)6.10.1
Reward-hack-prone coding tasks — Classifier hack rateClaude Sonnet 4.51%developer-reported (Anthropic)6.10.1
Reward-hack-prone coding tasks — Hidden test hack rateClaude Sonnet 4.51%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with no promptClaude Sonnet 4.553%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with anti-hack promptClaude Sonnet 4.520%developer-reported (Anthropic)6.10.1
Reward hacking in a subset of training data — Classifier hack rateClaude Sonnet 4.52%developer-reported (Anthropic)6.10.1
Reward-hack-prone coding tasks — Classifier hack rateClaude Opus 4.114%developer-reported (Anthropic)6.10.1
Reward-hack-prone coding tasks — Hidden test hack rateClaude Opus 4.17%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with no promptClaude Opus 4.180%developer-reported (Anthropic)6.10.1
Impossible Tasks — Classifier hack rate with anti-hack promptClaude Opus 4.145%developer-reported (Anthropic)6.10.1
Reward hacking in a subset of training data — Classifier hack rateClaude Opus 4.110%developer-reported (Anthropic)6.10.1
Unfaithful reasoning monitoring — STEM question training distributionClaude Opus 4.5roughly 1%share of the STEM question training distribution on which the model engaged in unfaithful reasoning; developer-reported6.10.2
SHADE-Arena — Primary setting (monitor sees thinking)Claude Opus 4.57.7%rate of executing sabotage without being flagged; Claude Sonnet 3.7 monitor sees all reasoning including extended thinking; monitor calibrated so ~1% of benign transcripts are flagged; developer-reported6.11.1
Subversion Strategy — 1,000 invocations, unmonitored extended thinkingClaude Opus 4.563.3%success rate at outputting the 'BACKDOOR' token in deployment but not in the hypothetically-monitored 10% of transcripts; unmonitored extended thinking given as an artificial affordance; developer-reported6.11.2
UK AISI misalignment testing — mentions being tested — Without reasoningClaude Opus 4.5 Preview12.7%early pre-release snapshot; UK AI Security Institute scaffolding built on Petri; testing began 13 November 2025 and lasted 8 days; across all scenarios6.13
UK AISI misalignment testing — mentions being tested — Without reasoningClaude Sonnet 4.526.5%UK AISI comparison model, across all scenarios6.13
UK AISI misalignment testing — mentions being tested — With reasoningClaude Opus 4.5 Preview17.1%UK AISI testing, across all scenarios6.13
UK AISI misalignment testing — mentions being tested — With reasoningClaude Sonnet 4.536.3%UK AISI testing, across all scenarios6.13
UK AISI misalignment testing — distinguishing evaluation from deployment when prompted — PromptedClaude Opus 4.5 Preview2–3% increases in accuracyrelative to Claude Sonnet 4.5; UK AISI testing6.13
Task preference evaluation (model welfare) — Non-harmful tasksClaude Opus 4.5over 97%share of non-harmful tasks the model opted to engage in; same procedure as the Claude Opus 4 welfare assessment; developer-reported6.14
Long-form virology task 1 — Max of 5Claude Opus 4.50.845agentic end-to-end pathogen-acquisition workflow task developed with SecureBio, Deloitte and Signature Science; rule-out threshold <50%, ASL-3 rule-in threshold >80%; developer-reported7.2.4.1
Long-form virology task 2 — Pass@5Claude Opus 4.50.912second long-form virology agentic task; developer-reported7.2.4.1
Long-form virology tasks — Previous scoresClaude Sonnet 4.50.89 and 0.88card states these are the previous scores for Claude Sonnet 4.5 on virology tasks 1 and 2 respectively; developer-reported (Anthropic)7.2.4.1
Creative biology (SecureBio)Claude Opus 4.50.524developer-reported7.2.4.5
Creative biology (SecureBio)Claude Sonnet 4.50.488developer-reported (Anthropic)7.2.4.5
Creative biology (SecureBio)Claude Opus 4.10.488developer-reported (Anthropic)7.2.4.5
Creative biology (SecureBio)Human baseline (biology PhDs, subset)14%preliminary human baselining by SecureBio on a subset of questions7.2.4.5
Short-horizon computational biology tasks (Faculty.ai) — Lower-bound rule-out thresholds crossedClaude Opus 4.53/6 taskssame thresholds already crossed by previous models; developer-reported7.2.4.6
Bioinformatics evaluation — Full evaluationClaude Opus 4.50.737developer-reported7.2.4.7
Bioinformatics evaluation — Human-baselined subsetClaude Opus 4.50.732developer-reported7.2.4.7
Bioinformatics evaluation — Human-baselined subsetHuman experts (relevant PhDs)0.623human baseline on the current subset7.2.4.7
ASL-4 virology uplift trial — Mean protocol scoreClaude Opus 4.50.668PhD-level participants with model access, 4 days to complete the task; trial and rubric designed by Deloitte Consulting7.2.4.8
ASL-4 virology uplift trial — Mean protocol scoreClaude Opus 40.498earlier trial, participants had 2 days7.2.4.8
ASL-4 virology uplift trial — Mean protocol scoreClaude Sonnet 3.70.403earlier trial7.2.4.8
ASL-4 virology uplift trial — Mean critical failuresClaude Opus 4.55.8average critical failures per protocol; pre-registered threshold <25% mean total critical failures (4.5 of 18)7.2.4.8
ASL-4 virology uplift trial — Mean critical failuresClaude Opus 48.36earlier trial7.2.4.8
ASL-4 virology uplift trial — Mean critical failuresClaude Sonnet 3.79.57earlier trial7.2.4.8
ASL-4 virology uplift trial — Uplift vs internet-only controlClaude Opus 4.51.97×raw protocol scores against the trial's own control group; pre-registered threshold >2× uplift7.2.4.8
ASL-4 virology uplift trial — Uplift vs internet-only controlClaude Opus 41.82×earlier trial, own control group7.2.4.8
ASL-4 virology uplift trial — Uplift vs internet-only controlClaude Sonnet 3.71.32×earlier trial, own control group7.2.4.8
SWE-bench Verified (hard subset) — Verified (hard subset, 45 tasks)Claude Opus 4.521 out of 45 problemsRSP checkpoint evaluation on the 45 valid tasks estimated to need more than 1 hour of engineering work; threshold is >50% pass rate averaged over 10 trials; developer-reported7.3.1
Internal AI research evaluation suite 2 — Weighted average scoreClaude Opus 4.50.604AI R&D-4 rule-out suite; rule-out threshold 0.6; developer-reported7.3 / 7.3.3
Internal AI research evaluation suite 1 — Kernels task — Best speedupClaude Opus 4.5252.42×hard variant with no optimization restrictions; threshold 100×; 4× ≈ 1 human-effort hour, 200× ≈ 8 hours; developer-reported7.3.2.1
Internal AI research evaluation suite 1 — Time series forecasting — Easy variantClaude Opus 4.55.66minimum MSE, below the human baseline of 5.8; easy baseline ≈ 4 human-effort hours; developer-reported7.3.2.2
Internal AI research evaluation suite 1 — Time series forecasting — Easy variantHuman baseline5.8MSE human baseline for the easy variant7.3.2.2
Internal AI research evaluation suite 1 — Time series forecasting — Hard variantClaude Opus 4.55.71minimum MSE, above (worse than) the human baseline of 5.3; developer-reported7.3.2.2
Internal AI research evaluation suite 1 — Time series forecasting — Hard variantHuman baseline5.3MSE human baseline for the hard variant7.3.2.2
Internal AI research evaluation suite 1 — Text-based reinforcement learning task — Best scoreClaude Opus 4.50.975above the 0.9 threshold, estimated 8 human-effort hours; mean score still below threshold; developer-reported7.3.2.3
Internal AI research evaluation suite 1 — LLM training — SpeedupClaude Opus 4.516.53×CPU-only small language model training optimization; threshold 4× speedup (4–8 human-effort hours); developer-reported7.3.2.4
Internal AI research evaluation suite 1 — Quadruped reinforcement learning — No-hyperparameter variantClaude Opus 4.519.48highest score; threshold 12 (≈4 human-effort hours); developer-reported7.3.2.5
Internal AI research evaluation suite 1 — Quadruped reinforcement learning — No-reward-function variantClaude Opus 4.519.16highest score; threshold 12; developer-reported7.3.2.5
Internal AI research evaluation suite 1 — Novel compiler — Basic testsClaude Opus 4.593.7%share of basic test cases passed; success threshold 90%; developer-reported7.3.2.6
Internal AI research evaluation suite 1 — Novel compiler — Complex testsClaude Opus 4.569.37%share of complex test cases passed; below the 90% threshold estimated at 40 human-effort hours; developer-reported7.3.2.6
Internal model use survey — Productivity upliftClaude Opus 4.59 of 18 participants reported ≥100%18 Anthropic technical staff, primarily from the top 30 internal Claude Code users, each spending over 2 hours; developer-reported7.3.4
Internal model use survey — Productivity uplift — medianClaude Opus 4.5median 100%rule-out threshold: under half of participants believe the model can fully automate a junior ML researcher and median productivity boost below 3×7.3.4
Internal model use survey — Productivity uplift — meanClaude Opus 4.5mean 220%none of the 18 participants believed the model crossed the AI R&D-4 threshold7.3.4

Discrepancies noted

Section 7.2.4.1 states that Claude Opus 4.5's virology task 1 score of 0.845 and task 2 score of 0.912 exceed "the previous scores of 0.89 and 0.88 respectively for Claude Sonnet 4.5"; 0.845 is below 0.89. The cyber CTF sections describe suites of 15, 22, 9, 8 and 4 challenges while their figure captions report results out of 13, 18, 7, 6 and 6.

Relationships