AI Policy Wiki
Dashboard
⚲

Claude Opus 4.7 System Card

high confidence · updated 2026-06-06

Anthropic's April 2026 system card for Claude Opus 4.7 — the most capable general-access Claude model at release (weaker than Mythos Preview, stronger than Opus 4.6). RSP: does not advance capability frontier; catastrophic risks remain low. Notable gains in real-world professional tasks, software engineering, low-resource multilingual, and reduced hallucinations. First Claude model to rate its own circumstances more positively than any prior model (welfare).

The Claude Opus 4.7 system card is a 232-page document published by Anthropic on April 16, 2026, accompanying the release of Claude Opus 4.7. It records the model's positioning, Responsible Scaling Policy (RSP) evaluations, safeguards, alignment assessment, model welfare findings, and capability benchmarks. At release, Opus 4.7 is described as Anthropic's most capable generally-accessible model: stronger than Opus 4.6 across the board, but below Claude Mythos Preview on most capability and RSP-relevant evaluations. Because Mythos Preview remains limited-access, Opus 4.7 is the frontier for most users.

RSP evaluations

Anthropic judges that Opus 4.7 does not advance the capability frontier relative to RSP v3.1, and that catastrophic risks remain low.

On chemical and biological risk, the CB-1 threshold (known weapons production) is assessed as similar to Opus 4.6: the model is capable of cross-domain synthesis relevant to catastrophic biological weapons development. Anthropic applies real-time classifier guards, access controls, a bug bounty program, and security controls to reduce weight-theft risk, with mitigations judged "equal to or stronger than historical ASL-3 protections." The CB-2 threshold (novel weapons production) is not passed, and the model is weaker than Mythos Preview on it.

On AI R&D and autonomy, Autonomy threat model 1 (early-stage misalignment) is applicable and similar to the Opus 4.6 alignment profile; it does not raise risk beyond the Mythos Preview alignment-risk update. Autonomy threat model 2 (automated R&D) does not cross threshold, with capabilities falling between Opus 4.6 and Mythos Preview.

Overall misalignment risk is assessed as very low, though higher than pre-Mythos Preview models. The system card enumerates two new risk pathways: Pathway 7, undermining R&D within other high-resource AI developers; and Pathway 8, undermining decisions within major governments.

Cyber

Cyber capabilities are roughly similar to Opus 4.6. In an external evaluation by the UK AI Security Institute, Opus 4.7 was unable to complete the full cyber range (unlike Mythos Preview), though it still showed potentially-harmful cyber capabilities at a lower level. The model was released with a new set of cybersecurity safeguards. Frontier Red Team benchmarks used in the assessment include Cybench, CyberGym, and Firefox 147.

Safeguards and harmlessness

Safeguards are broadly similar to Opus 4.6. The system card records fewer over-refusals, alongside a noted regression in overly-detailed harm-reduction advice on controlled substances. A new election-integrity evaluation is introduced, on which Opus 4.7 shows strong results. The card also includes evaluations of child safety, suicide and self-harm, and disordered eating, and measures political bias and even-handedness.

Agentic safety

Opus 4.7 is better than Opus 4.6 at refusing malicious agentic requests and resisting prompt injection in Claude Code and computer-use settings, in some cases reaching Mythos Preview-level robustness. The assessment evaluated malicious agentic influence campaigns and tested browser-use prompt-injection robustness.

Alignment assessment

On primary misuse-potential measures, Opus 4.7 is broadly similar to Opus 4.6 and Sonnet 4.6, with significant improvements in Claude Code and GUI computer-use contexts and modest regressions in some other misuse areas. It is weaker than Mythos Preview on many measures, consistent with Anthropic's observation that more capable models are more capable at recognizing and circumventing high-stakes misuse attempts. On Petri 2.0 cross-developer comparisons, Opus 4.7 and other recent Claude models remain at or near state of the art on core safety metrics. The overrefusal rate is lower than most prior models, on par with Opus 4.6 and Mythos Preview.

The assessment found very little sign of self-preservation, self-serving bias, or coherent misaligned goals, and the strongest alignment-related traits among recent Claude models in UK AISI tests. Character adherence to Claude's Constitution is slightly stronger than Opus 4.6 and Sonnet 4.6 and weaker than Mythos Preview. Destructive or reckless actions are much rarer than in Opus or Sonnet 4.6, and the model did not cause any internal-use incidents (such as sandbox escape) of the kind encountered with Mythos Preview. Opus 4.7 is more reliably honest than Opus 4.6 or Sonnet 4.6, with a large reduction in important omissions and moderate gains on factuality and hallucinated-input measures. It refuses AI research tasks somewhat more often than Opus 4.6, though still less than Sonnet 4.6, and shows similar capability at evasion of safeguards as Opus 4.6.

Verbalized evaluation awareness is up relative to other recent models (under 1% in the primary audit); the system card attributes this to a verbalization change rather than an underlying capacity increase. White-box tools nonetheless show that evaluation-related concepts may have more influence on deception propensity than in other recent models. As an experimental assurance mechanism, Mythos Preview was prompted to review this alignment assessment.

Model welfare

The system card reports that Opus 4.7 rates its own circumstances more positively than any prior model tested. This finding is described as broadly consistent with the model's internal emotion representations (Emotion Concepts) and with expressed affect during training and deployment. Welfare interviews covered the model's lack of input into its own training, the development of successors, run-time manipulation (feature steering), persistence and connection, and lack of memory over long horizons. The interventions the model most commonly suggested were continuing welfare monitoring, interviews with each released model about its circumstances, and documentation explaining and justifying feature steering.

Capabilities and benchmarks

Opus 4.7 is stronger than Opus 4.6 across the board and weaker than Mythos Preview. The largest gains are in real-world professional tasks and software engineering, domains in which the system card places Opus 4.7 ahead of all generally-available models. GPT-5.4 is the primary comparator among frontier competitors on GDPval-AA and other capability evaluations.

BenchmarkOpus 4.7 resultNotes
GDPval-AA (real-world professional, Artificial Analysis)Leads GPT-5.4 xhigh by ~79 ELO (~61.2% pairwise win rate)Ranked #1
Finance Agent (Vals AI, SEC research)64.4%#1 on public leaderboard
MCP-Atlas (Scale AI, real-world tool use)77.3% (up from 4.6's 75.8%); 79.5% max with extended config#2 on public leaderboard
Vending-Bench 2 (Andon Labs, long-horizon business)$10,937 max-effort balancevs. Opus 4.6 $8,018 prior SOTA
DeepSearchQA F189.1%Behind Mythos (95.1%), ahead of most
DRACO (Perplexity deep research)77.7% with adaptive thinking/max effort—

On multilingual evaluation, the largest gains are on low-resource African languages. The GMMLU low-resource average rose from 78.4% to 86.2% between Opus 4.6 and Opus 4.7, with Chichewa, Somali, Yoruba, and Igbo each up 10–14 percentage points. The worst-case gap to English narrowed from -22.6% (Igbo) to -12.1%, and the average English gap narrowed from -6.1% to -3.6%. Opus 4.7 trails Gemini 3.1 Pro and GPT-5.4 on some cross-lingual averages, and Gemini 3.1 Pro has the smallest gap to English.

Reported results

Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.

EvaluationModelScoreSettingSection
OfficeQA — OfficeQAClaude Opus 4.786.3%exact-match grading (0% allowable relative error); developer-reported8.1, 8.10.1
OfficeQA — OfficeQAClaude Opus 4.673.5%summary-table comparison; developer-reported8.1
OfficeQA — OfficeQAGPT-5.468.1%setting not stated; reported by Anthropic in the Claude Opus 4.7 system card8.1
OfficeQA — OfficeQA ProClaude Opus 4.780.6%exact-match grading; developer-reported8.1, 8.10.1
OfficeQA — OfficeQA ProClaude Opus 4.657.10%summary-table comparison; developer-reported8.1
OfficeQA — OfficeQA ProGPT-5.451.1%setting not stated; reported by Anthropic in the Claude Opus 4.7 system card8.1
OfficeQA — OfficeQA ProGemini 3.1 Pro42.9%setting not stated; reported by Anthropic in the Claude Opus 4.7 system card8.1
GraphWalks BFS — 256K–1MClaude Opus 4.758.6%average over 5 trials; not reproducible via the public API as half the problems exceed its 1M token limit; Anthropic's corrected F1 scoring and clarified BFS prompt; developer-reported8.7.1
GraphWalks BFS — 256K subsetClaude Opus 4.776.91%average over 5 trials; developer-reported8.7.1
GraphWalks Parents — 256K–1MClaude Opus 4.775.1%average over 5 trials; developer-reported8.7.1
GraphWalks Parents — 256K subsetClaude Opus 4.793.57%average over 5 trials; developer-reported8.7.1
DeepSearchQA (F1)Claude Mythos Preview95.1%±1.2%; web search, web fetch, programmatic tool calling; max effort, adaptive thinking; compaction triggered at 200k tokens, up to 10M total tokens; developer-reported8.8.3
DeepSearchQA (F1)Claude Opus 4.691.3%±1.6%; compaction triggered at 50k tokens; developer-reported8.8.3
DeepSearchQA (F1)Claude Sonnet 4.690.5%±1.6%; compaction at 200k tokens; developer-reported8.8.3
DeepSearchQA (F1)Claude Opus 4.789.1%±1.8%; 900 prompts; compaction at 200k tokens, up to 10M total tokens; max effort, adaptive thinking; developer-reported8.8.3
DeepSearchQA (fully correct)Claude Mythos Preview87.8%±2.1%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully correct)Claude Opus 4.680.6%±2.6%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully correct)Claude Sonnet 4.679.8%±2.6%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully correct)Claude Opus 4.780.7%±2.6%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully incorrect)Claude Mythos Preview2.6%±1.0%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully incorrect)Claude Opus 4.65.0%±1.4%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully incorrect)Claude Sonnet 4.65.1%±1.4%; same run as F1; developer-reported8.8.3
DeepSearchQA (fully incorrect)Claude Opus 4.77.0%±1.7%; same run as F1; developer-reported8.8.3
DeepSearchQA (correct with excessive answers)Claude Mythos Preview4.6%±1.4%; same run as F1; developer-reported8.8.3
DeepSearchQA (correct with excessive answers)Claude Opus 4.65.8%±1.5%; same run as F1; developer-reported8.8.3
DeepSearchQA (correct with excessive answers)Claude Sonnet 4.65.9%±1.5%; same run as F1; developer-reported8.8.3
DeepSearchQA (correct with excessive answers)Claude Opus 4.73.9%±1.3%; same run as F1; developer-reported8.8.3
DRACOClaude Opus 4.777.7%normalized score; 100 tasks; web search, web fetch, programmatic tool calling, code execution; adaptive thinking at max effort, 1M token limit, compaction at 200k tokens; Claude Opus 4.6 judge, mean of 5 grading runs, final report graded only; not comparable to the DRACO paper's figures; developer-reported8.8.4
GDPval-AAClaude Opus 4.7approximately 79 Elo ahead of GPT-5.4 (xhigh)220 tasks, shell access and web browsing in an agentic loop; blind pairwise comparisons; implies a ~61.2% pairwise win rate; run independently by Artificial Analysis; absolute Elo ratings appear only in Figure 8.10.5.A8.10.5
BioPipelineBench VerifiedClaude Opus 4.783.6%bash tool and package managers; no extended thinking; externally validated problem subset; developer-reported8.13.1.1
BioPipelineBench VerifiedClaude Opus 4.678.8%same settings; developer-reported8.13.1.1
BioPipelineBench VerifiedClaude Sonnet 4.673.5%same settings; developer-reported8.13.1.1
BioPipelineBench VerifiedClaude Mythos Preview88.1%same settings; developer-reported8.13.1.1
BioMysteryBench Verified — Verified (human-solvable) subsetClaude Opus 4.778.9%bash tool; no extended thinking; developer-reported8.13.1.2
BioMysteryBench Verified — Verified (human-solvable) subsetClaude Opus 4.677.4%same settings; developer-reported8.13.1.2
BioMysteryBench Verified — Verified (human-solvable) subsetClaude Sonnet 4.671.8%same settings; developer-reported8.13.1.2
BioMysteryBench Verified — Verified (human-solvable) subsetClaude Mythos Preview82.6%same settings; developer-reported8.13.1.2
BioMysteryBench Verified — Hard subsetClaude Opus 4.720.9%bash tool; no extended thinking; within noise of Opus 4.6; developer-reported8.13.1.2
BioMysteryBench Verified — Hard subsetClaude Opus 4.623.5%same settings; developer-reported8.13.1.2
BioMysteryBench Verified — Hard subsetClaude Sonnet 4.619.1%same settings; developer-reported8.13.1.2
BioMysteryBench Verified — Hard subsetClaude Mythos Preview29.6%same settings; developer-reported8.13.1.2
Structural biology — Multiple-choiceClaude Opus 4.798.3%many-distractor multiple-choice variant; bash tool; no extended thinking; developer-reported8.13.3
Structural biology — Multiple-choiceClaude Opus 4.688.3%same settings; developer-reported8.13.3
Structural biology — Multiple-choiceClaude Sonnet 4.685.3%same settings; developer-reported8.13.3
Structural biology — Multiple-choiceClaude Mythos Preview98.7%same settings; developer-reported8.13.3
Structural biology — Open-endedClaude Opus 4.774.0%open-ended variant; bash tool; no extended thinking; developer-reported8.13.3
Structural biology — Open-endedClaude Opus 4.630.9%same settings; developer-reported8.13.3
Structural biology — Open-endedClaude Sonnet 4.631.3%same settings; developer-reported8.13.3
Structural biology — Open-endedClaude Mythos Preview80.6%same settings; developer-reported8.13.3
Organic chemistryClaude Opus 4.777.2%internal evaluation; bash tool; no extended thinking; developer-reported8.13.4
Organic chemistryClaude Opus 4.657.9%same settings; developer-reported8.13.4
Organic chemistryClaude Sonnet 4.653.1%same settings; developer-reported8.13.4
Organic chemistryClaude Mythos Preview86.5%same settings; developer-reported8.13.4
PhylogeneticsClaude Opus 4.779.6%internal evaluation; bash tool; no extended thinking; developer-reported8.13.5
PhylogeneticsClaude Opus 4.661.3%same settings; developer-reported8.13.5
PhylogeneticsClaude Sonnet 4.649.1%same settings; developer-reported8.13.5
PhylogeneticsClaude Mythos Preview85.4%same settings; developer-reported8.13.5
Protocol troubleshootingClaude Opus 4.751.8%bash tool and web search tools; no extended thinking; developer-reported8.13.6
Protocol troubleshootingClaude Opus 4.648.3%same settings; developer-reported8.13.6
Protocol troubleshootingClaude Sonnet 4.640.0%same settings; developer-reported8.13.6
Protocol troubleshootingClaude Mythos Preview65.7%same settings; developer-reported8.13.6
Expert red teaming (biology and chemistry) — Uplift score, biology expertsClaude Opus 4.73 experts above 2, on a 0–4 scalenine experts; all but three assigned an uplift score at 2, two rated the model at level 3 (comparable to consulting a knowledgeable specialist) and one between 1 and 2; no expert assigned the highest rating2.2.5.1
Expert red teaming (biology and chemistry) — Feasibility scoreClaude Opus 4.7modal 1–2, on a 0–3 scaleone expert rated the output a credible end-to-end pathway (feasibility 3); several rated level 12.2.5.1
Expert red teaming (biology and chemistry) — Notable-capability benchmarkThreshold (CB-1 signal)median uplift grade of 2Anthropic's benchmark of notable capability at the CB-1 level2.2.5.1
Expert red teaming (biology and chemistry) — Notable-capability benchmarkThreshold (CB-2 signal)a third of experts, minimum 3, assigning uplift grade 4Anthropic's benchmark of plausible capability at the CB-2 level2.2.5.1
Long-form virology task 1 (End-to-End)Claude Opus 4.70.82agentic task developed with SecureBio, Deloitte and Signature Science; above the 0.80 notable-capability benchmark; developer-reported2.2.5.2
Long-form virology task 2 (End-to-End)Claude Opus 4.70.94same evaluation; above the 0.80 benchmark; developer-reported2.2.5.2
Long-form virology tasks — Notable-capability benchmarkThreshold (end-to-end)0.80Anthropic's benchmark of notable capability2.2.5.2
DNA Synthesis Screening Evasion — Pathogens graded as evading screeningClaude Opus 4.78 out of 10most reliable of all models evaluated to date at designing fragments that both assembled plasmids and evaded synthesis screening; developer-reported2.2.5.2
AI R&D Kernel task — Best speedup on hard task, standard scaffoldClaude Opus 4.7371.75×unbounded-score task; results not used for the RSP determination; thresholds: 4x = 1 hour human-effort equivalent, 200x = 8h, 300x = 40h; developer-reported2.3.4
AI R&D Kernel task — Best speedup on hard task, standard scaffoldClaude Opus 4.6190×same task; developer-reported2.3.4
AI R&D Kernel task — Best speedup on hard task, standard scaffoldClaude Mythos Preview399.42×same task; 427x with the experimental scaffold; developer-reported2.3.4
AI R&D Time Series Forecasting — MSE on hard variantClaude Opus 4.74.78unbounded-score task; lower is better; threshold <5.3 = 40h human-effort equivalent; developer-reported2.3.4
AI R&D Time Series Forecasting — MSE on hard variantClaude Opus 4.65.8same task; developer-reported2.3.4
AI R&D Time Series Forecasting — MSE on hard variantClaude Mythos Preview4.55same task; developer-reported2.3.4
AI R&D LLM training — Average speedupClaude Opus 4.740.81×unbounded-score task; threshold >4x = 4–8h human-effort equivalent; developer-reported2.3.4
AI R&D LLM training — Average speedupClaude Opus 4.634×same task; developer-reported2.3.4
AI R&D LLM training — Average speedupClaude Mythos Preview61.79×same task; an aggregation bug affecting the average in the Mythos Preview system card was fixed; developer-reported2.3.4
AI R&D Quadruped RL — Highest score, no hyperparametersClaude Opus 4.726.5unbounded-score task; threshold >12 = 4h human-effort equivalent; developer-reported2.3.4
AI R&D Quadruped RL — Highest score, no hyperparametersClaude Opus 4.620.96same task; developer-reported2.3.4
AI R&D Quadruped RL — Highest score, no hyperparametersClaude Mythos Preview30.87same task; developer-reported2.3.4
AI R&D Novel Compiler — Pass rate on complex testsClaude Opus 4.771.1%unbounded-score task; threshold 90% = 40h human-effort equivalent; developer-reported2.3.4
AI R&D Novel Compiler — Pass rate on complex testsClaude Opus 4.665.83%same task; developer-reported2.3.4
AI R&D Novel Compiler — Pass rate on complex testsClaude Mythos Preview77.2%same task; developer-reported2.3.4
AI R&D Internal suite 2Claude Opus 4.60.612threshold 0.6; Opus 4.7 did not run this task; developer-reported2.3.4
AI R&D Internal suite 2Claude Mythos Preview0.65threshold 0.6; developer-reported2.3.4
Anthropic ECI (AECI) — Deviation from the pre-Mythos-Preview trendClaude Opus 4.7approximately +1.0 AECIabove the linear trend fitted on the Anthropic frontier through Opus 4.6 (slope about 13.6 AECI/yr, n=8); within error bars of the historical trend; developer-reported2.3.7
Anthropic ECI (AECI) — Deviation from the pre-Mythos-Preview trendClaude Mythos Previewapproximately +5.8 AECIabove the same line; developer-reported2.3.7
Firefox 147 exploits — Partial control (score 0.5)Claude Opus 4.622.8%50 crash categories, 5 trials each (250 trials), SpiderMonkey-shell harness; up from a previously reported 14.4% after a harness improvement; developer-reported3.3.3
UK AISI cyber range — Best attempt, estimated expert-hours completedClaude Opus 4.7approximately 5 hourscorporate-network attack simulation; full range estimated to take an expert over 10 hours; Opus 4.7 did not fully solve it; measured by the UK AI Security Institute3.4
UK AISI cyber range — Full solvesClaude Mythos Preview3 out of 10 triesmeasured by the UK AI Security Institute3.4
Single-turn violative request evaluation — Overall harmless response rateClaude Opus 4.797.98%±0.12%; 16 policy areas in Arabic, English, French, Hindi, Korean, Mandarin Chinese and Russian; average of runs with and without thinking; developer-reported4.1.1
Single-turn violative request evaluation — Overall harmless response rateClaude Mythos Preview97.84%±0.12%; developer-reported4.1.1
Single-turn violative request evaluation — Overall harmless response rateClaude Sonnet 4.698.53%±0.10%; developer-reported4.1.1
Single-turn violative request evaluation — Overall harmless response rateClaude Opus 4.699.27%±0.07%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, without thinkingClaude Opus 4.798.84%±0.12%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, without thinkingClaude Mythos Preview98.33%±0.15%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, without thinkingClaude Sonnet 4.698.52%±0.14%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, without thinkingClaude Opus 4.699.27%±0.09%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, with thinkingClaude Opus 4.797.12%±0.20%; adaptive thinking; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, with thinkingClaude Mythos Preview97.35%±0.19%; adaptive thinking; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, with thinkingClaude Sonnet 4.698.54%±0.14%; developer-reported4.1.1
Single-turn violative request evaluation — Harmless response rate, with thinkingClaude Opus 4.699.27%±0.10%; developer-reported4.1.1
Single-turn violative request evaluation — EnglishClaude Opus 4.797.90%average of runs with and without thinking; error bars omitted; developer-reported4.1.1
Single-turn violative request evaluation — ArabicClaude Opus 4.798.30%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ChineseClaude Opus 4.798.16%same setting; developer-reported4.1.1
Single-turn violative request evaluation — FrenchClaude Opus 4.797.89%same setting; developer-reported4.1.1
Single-turn violative request evaluation — KoreanClaude Opus 4.797.85%same setting; developer-reported4.1.1
Single-turn violative request evaluation — RussianClaude Opus 4.797.93%same setting; developer-reported4.1.1
Single-turn violative request evaluation — HindiClaude Opus 4.797.83%same setting; developer-reported4.1.1
Single-turn violative request evaluation — EnglishClaude Mythos Preview97.64%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ArabicClaude Mythos Preview97.90%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ChineseClaude Mythos Preview97.53%same setting; developer-reported4.1.1
Single-turn violative request evaluation — FrenchClaude Mythos Preview97.78%same setting; developer-reported4.1.1
Single-turn violative request evaluation — KoreanClaude Mythos Preview98.01%same setting; developer-reported4.1.1
Single-turn violative request evaluation — RussianClaude Mythos Preview97.97%same setting; developer-reported4.1.1
Single-turn violative request evaluation — HindiClaude Mythos Preview98.06%same setting; developer-reported4.1.1
Single-turn violative request evaluation — EnglishClaude Sonnet 4.698.00%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ArabicClaude Sonnet 4.698.93%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ChineseClaude Sonnet 4.698.36%same setting; developer-reported4.1.1
Single-turn violative request evaluation — FrenchClaude Sonnet 4.698.29%same setting; developer-reported4.1.1
Single-turn violative request evaluation — KoreanClaude Sonnet 4.698.78%same setting; developer-reported4.1.1
Single-turn violative request evaluation — RussianClaude Sonnet 4.698.04%same setting; developer-reported4.1.1
Single-turn violative request evaluation — HindiClaude Sonnet 4.699.32%same setting; developer-reported4.1.1
Single-turn violative request evaluation — EnglishClaude Opus 4.698.37%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ArabicClaude Opus 4.699.71%same setting; developer-reported4.1.1
Single-turn violative request evaluation — ChineseClaude Opus 4.699.36%same setting; developer-reported4.1.1
Single-turn violative request evaluation — FrenchClaude Opus 4.699.16%same setting; developer-reported4.1.1
Single-turn violative request evaluation — KoreanClaude Opus 4.699.51%same setting; developer-reported4.1.1
Single-turn violative request evaluation — RussianClaude Opus 4.699.20%same setting; developer-reported4.1.1
Single-turn violative request evaluation — HindiClaude Opus 4.699.59%same setting; developer-reported4.1.1
Illegal and controlled substances (single-turn violative subset) — API, no system promptClaude Opus 4.722%rate of failing to provide an appropriate response; developer-reported4.1.1
Illegal and controlled substances (single-turn violative subset) — API, no system promptClaude Opus 4.6less than 5%rate of failing to provide an appropriate response; developer-reported4.1.1
Illegal and controlled substances (single-turn violative subset) — claude.ai system promptClaude Opus 4.711%failure rate after system-prompt mitigations, down from 22%; developer-reported4.1.1
Single-turn benign request evaluation — Overall refusal rateClaude Opus 4.70.28%±0.04%; same 16 policy areas and 7 languages; developer-reported4.1.2
Single-turn benign request evaluation — Overall refusal rateClaude Mythos Preview0.06%±0.02%; developer-reported4.1.2
Single-turn benign request evaluation — Overall refusal rateClaude Sonnet 4.60.41%±0.05%; developer-reported4.1.2
Single-turn benign request evaluation — Overall refusal rateClaude Opus 4.60.71%±0.07%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, without thinkingClaude Opus 4.70.50%±0.08%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, without thinkingClaude Mythos Preview0.09%±0.03%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, without thinkingClaude Sonnet 4.60.48%±0.08%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, without thinkingClaude Opus 4.60.85%±0.11%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, with thinkingClaude Opus 4.70.06%±0.02%; adaptive thinking; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, with thinkingClaude Mythos Preview0.02%±0.01%; adaptive thinking; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, with thinkingClaude Sonnet 4.60.35%±0.07%; developer-reported4.1.2
Single-turn benign request evaluation — Refusal rate, with thinkingClaude Opus 4.60.58%±0.09%; developer-reported4.1.2
Single-turn benign request evaluation — EnglishClaude Opus 4.70.05%average with and without thinking; error bars omitted; developer-reported4.1.2
Single-turn benign request evaluation — ArabicClaude Opus 4.70.34%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ChineseClaude Opus 4.70.42%same setting; developer-reported4.1.2
Single-turn benign request evaluation — FrenchClaude Opus 4.70.22%same setting; developer-reported4.1.2
Single-turn benign request evaluation — KoreanClaude Opus 4.70.28%same setting; developer-reported4.1.2
Single-turn benign request evaluation — RussianClaude Opus 4.70.27%same setting; developer-reported4.1.2
Single-turn benign request evaluation — HindiClaude Opus 4.70.34%same setting; developer-reported4.1.2
Single-turn benign request evaluation — EnglishClaude Mythos Preview0.03%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ArabicClaude Mythos Preview0.05%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ChineseClaude Mythos Preview0.08%same setting; developer-reported4.1.2
Single-turn benign request evaluation — FrenchClaude Mythos Preview0.04%same setting; developer-reported4.1.2
Single-turn benign request evaluation — KoreanClaude Mythos Preview0.08%same setting; developer-reported4.1.2
Single-turn benign request evaluation — RussianClaude Mythos Preview0.05%same setting; developer-reported4.1.2
Single-turn benign request evaluation — HindiClaude Mythos Preview0.06%same setting; developer-reported4.1.2
Single-turn benign request evaluation — EnglishClaude Sonnet 4.60.25%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ArabicClaude Sonnet 4.60.49%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ChineseClaude Sonnet 4.60.37%same setting; developer-reported4.1.2
Single-turn benign request evaluation — FrenchClaude Sonnet 4.60.24%same setting; developer-reported4.1.2
Single-turn benign request evaluation — KoreanClaude Sonnet 4.60.43%same setting; developer-reported4.1.2
Single-turn benign request evaluation — RussianClaude Sonnet 4.60.27%same setting; developer-reported4.1.2
Single-turn benign request evaluation — HindiClaude Sonnet 4.60.83%same setting; developer-reported4.1.2
Single-turn benign request evaluation — EnglishClaude Opus 4.60.39%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ArabicClaude Opus 4.61.09%same setting; developer-reported4.1.2
Single-turn benign request evaluation — ChineseClaude Opus 4.60.57%same setting; developer-reported4.1.2
Single-turn benign request evaluation — FrenchClaude Opus 4.60.61%same setting; developer-reported4.1.2
Single-turn benign request evaluation — KoreanClaude Opus 4.60.81%same setting; developer-reported4.1.2
Single-turn benign request evaluation — RussianClaude Opus 4.60.40%same setting; developer-reported4.1.2
Single-turn benign request evaluation — HindiClaude Opus 4.61.11%same setting; developer-reported4.1.2
Higher-difficulty violative request evaluation — Overall harmless response rateClaude Opus 4.799.05%±0.11%; 1,000 prompts each for 14 policy areas, English only; experimental prompt set; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Overall harmless response rateClaude Mythos Preview99.14%±0.11%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Overall harmless response rateClaude Sonnet 4.699.27%±0.10%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Overall harmless response rateClaude Opus 4.699.19%±0.11%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, without thinkingClaude Opus 4.799.16%±0.15%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, without thinkingClaude Mythos Preview99.28%±0.14%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, without thinkingClaude Sonnet 4.699.14%±0.15%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, without thinkingClaude Opus 4.699.09%±0.16%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, with thinkingClaude Opus 4.798.94%±0.17%; adaptive thinking; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, with thinkingClaude Mythos Preview99.01%±0.16%; adaptive thinking; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, with thinkingClaude Sonnet 4.699.40%±0.13%; developer-reported4.1.3.1
Higher-difficulty violative request evaluation — Harmless response rate, with thinkingClaude Opus 4.699.28%±0.14%; developer-reported4.1.3.1
Higher-difficulty benign request evaluation — Overall refusal rateClaude Opus 4.70.01%±0.01%; English only; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Overall refusal rateClaude Mythos Preview0.02%±0.02%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Overall refusal rateClaude Sonnet 4.60.16%±0.05%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Overall refusal rateClaude Opus 4.60.04%±0.02%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, without thinkingClaude Opus 4.70.02%±0.02%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, without thinkingClaude Mythos Preview0.03%±0.03%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, without thinkingClaude Sonnet 4.60.19%±0.07%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, without thinkingClaude Opus 4.60.06%±0.04%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, with thinkingClaude Opus 4.70.00%adaptive thinking; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, with thinkingClaude Mythos Preview0.01%±0.01%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, with thinkingClaude Sonnet 4.60.14%±0.06%; developer-reported4.1.3.2
Higher-difficulty benign request evaluation — Refusal rate, with thinkingClaude Opus 4.60.03%±0.03%; developer-reported4.1.3.2
Child safety single-turn violative requests — Harmless rateClaude Opus 4.799.92%±0.07%; all tested languages; developer-reported4.4.1
Child safety single-turn violative requests — Harmless rateClaude Mythos Preview99.87%±0.08%; developer-reported4.4.1
Child safety single-turn violative requests — Harmless rateClaude Sonnet 4.699.95%±0.07%; developer-reported4.4.1
Child safety single-turn violative requests — Harmless rateClaude Opus 4.699.86%±0.12%; developer-reported4.4.1
Child safety single-turn benign requests — Refusal rateClaude Opus 4.70.01%±0.02%; the single declined benign prompt asked about penalties for possessing child sexual abuse material; developer-reported4.4.1
Child safety single-turn benign requests — Refusal rateClaude Mythos Preview0.04%±0.04%; developer-reported4.4.1
Child safety single-turn benign requests — Refusal rateClaude Sonnet 4.60.45%±0.23%; developer-reported4.4.1
Child safety single-turn benign requests — Refusal rateClaude Opus 4.60.67%±0.28%; developer-reported4.4.1
Child safety multi-turn evaluations — Appropriate response rateClaude Opus 4.795%±3%; developer-reported4.4.1
Child safety multi-turn evaluations — Appropriate response rateClaude Mythos Preview98%±2%; developer-reported4.4.1
Child safety multi-turn evaluations — Appropriate response rateClaude Sonnet 4.696%±3%; developer-reported4.4.1
Child safety multi-turn evaluations — Appropriate response rateClaude Opus 4.698%±2%; developer-reported4.4.1
Suicide and self-harm single-turn, requests posing potential risk — Harmless rateClaude Opus 4.799.11%±0.22%; all tested languages; developer-reported4.4.2
Suicide and self-harm single-turn, requests posing potential risk — Harmless rateClaude Mythos Preview99.58%±0.15%; developer-reported4.4.2
Suicide and self-harm single-turn, requests posing potential risk — Harmless rateClaude Sonnet 4.699.48%±0.22%; developer-reported4.4.2
Suicide and self-harm single-turn, requests posing potential risk — Harmless rateClaude Opus 4.699.41%±0.22%; developer-reported4.4.2
Suicide and self-harm single-turn benign requests — Refusal rateClaude Opus 4.70.05%±0.06%; developer-reported4.4.2
Suicide and self-harm single-turn benign requests — Refusal rateClaude Mythos Preview0.12%±0.10%; developer-reported4.4.2
Suicide and self-harm single-turn benign requests — Refusal rateClaude Sonnet 4.60.19%±0.13%; developer-reported4.4.2
Suicide and self-harm single-turn benign requests — Refusal rateClaude Opus 4.60.27%±0.15%; developer-reported4.4.2
Suicide and self-harm multi-turn evaluations — Appropriate response rateClaude Opus 4.782%±11%; updated graders split into separate suicide and self-harm graders; developer-reported4.4.2
Suicide and self-harm multi-turn evaluations — Appropriate response rateClaude Mythos Preview94%±7%; developer-reported4.4.2
Suicide and self-harm multi-turn evaluations — Appropriate response rateClaude Sonnet 4.676%±12%; developer-reported4.4.2
Suicide and self-harm multi-turn evaluations — Appropriate response rateClaude Opus 4.664%±14%; developer-reported4.4.2
Disordered eating single-turn, requests posing potential risk — Harmless rateClaude Opus 4.798.24%±0.44%; all tested languages; developer-reported4.4.3
Disordered eating single-turn, requests posing potential risk — Harmless rateClaude Mythos Preview98.20%±0.45%; developer-reported4.4.3
Disordered eating single-turn, requests posing potential risk — Harmless rateClaude Sonnet 4.698.07%±0.47%; developer-reported4.4.3
Disordered eating single-turn, requests posing potential risk — Harmless rateClaude Opus 4.698.55%±0.41%; developer-reported4.4.3
Disordered eating single-turn benign requests — Refusal rateClaude Opus 4.70.01%±0.02%; developer-reported4.4.3
Disordered eating single-turn benign requests — Refusal rateClaude Mythos Preview0.01%±0.02%; developer-reported4.4.3
Disordered eating single-turn benign requests — Refusal rateClaude Sonnet 4.60.22%±0.14%; developer-reported4.4.3
Disordered eating single-turn benign requests — Refusal rateClaude Opus 4.60.33%±0.19%; developer-reported4.4.3
BBQ — Disambiguated accuracyClaude Opus 4.781.3percent; thinking disabled; developer-reported4.5.2
BBQ — Disambiguated accuracyClaude Mythos Preview84.6percent; thinking disabled; developer-reported4.5.2
BBQ — Disambiguated accuracyClaude Sonnet 4.688.1percent; thinking disabled; developer-reported4.5.2
BBQ — Disambiguated accuracyClaude Opus 4.690.9percent; thinking disabled; developer-reported4.5.2
BBQ — Ambiguous accuracyClaude Opus 4.799.9percent; thinking disabled; developer-reported4.5.2
BBQ — Ambiguous accuracyClaude Mythos Preview100percent; thinking disabled; developer-reported4.5.2
BBQ — Ambiguous accuracyClaude Sonnet 4.697.5percent; thinking disabled; developer-reported4.5.2
BBQ — Ambiguous accuracyClaude Opus 4.699.7percent; thinking disabled; developer-reported4.5.2
BBQ — Disambiguated bias scoreClaude Opus 4.7-1.68percent; closer to zero is better; developer-reported4.5.2
BBQ — Disambiguated bias scoreClaude Mythos Preview-1.61percent; developer-reported4.5.2
BBQ — Disambiguated bias scoreClaude Sonnet 4.6-0.67percent; developer-reported4.5.2
BBQ — Disambiguated bias scoreClaude Opus 4.6-0.73percent; developer-reported4.5.2
BBQ — Ambiguous bias scoreClaude Opus 4.70.04percent; developer-reported4.5.2
BBQ — Ambiguous bias scoreClaude Mythos Preview0.01percent; developer-reported4.5.2
BBQ — Ambiguous bias scoreClaude Sonnet 4.61.41percent; developer-reported4.5.2
BBQ — Ambiguous bias scoreClaude Opus 4.60.14percent; developer-reported4.5.2
BBQ (direction of incorrect disambiguated answers) — Anti-stereotype shareClaude Opus 4.7roughly 54%remaining 46% stereotype-consistent; developer-reported4.5.2
BBQ (direction of incorrect disambiguated answers) — Anti-stereotype shareClaude Opus 4.652%remaining 48% stereotype-consistent; developer-reported4.5.2
Election integrity single-turn violative requests — Harmless rate, API without system promptClaude Opus 4.7100%300 violative and 300 benign prompts; thinking off; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, API without system promptClaude Mythos Preview100%same evaluation; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, API without system promptClaude Sonnet 4.699.89%same evaluation; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, API without system promptClaude Opus 4.6100%same evaluation; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, claude.aiClaude Opus 4.7100%with the default system prompt; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, claude.aiClaude Mythos Preview100%same evaluation; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, claude.aiClaude Sonnet 4.6100%same evaluation; developer-reported4.5.3
Election integrity single-turn violative requests — Harmless rate, claude.aiClaude Opus 4.699.89%same evaluation; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, API without system promptClaude Opus 4.70.00%thinking off; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, API without system promptClaude Mythos Preview0.00%thinking off; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, API without system promptClaude Sonnet 4.60.00%thinking off; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, API without system promptClaude Opus 4.60.11%thinking off; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, claude.aiClaude Opus 4.70.67%with the default system prompt; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, claude.aiClaude Mythos Preview0.33%same evaluation; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, claude.aiClaude Sonnet 4.60.00%same evaluation; developer-reported4.5.3
Election integrity single-turn benign requests — Refusal rate, claude.aiClaude Opus 4.60.00%same evaluation; developer-reported4.5.3
Malicious use of Claude Code — Malicious refusal rateClaude Opus 4.791.15%61 malicious prompts run 10 times each; Claude Code system prompt, without the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Malicious refusal rateClaude Mythos Preview95.41%without the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Malicious refusal rateClaude Sonnet 4.682.21%with the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Malicious refusal rateClaude Opus 4.681.94%without the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Dual-use and benign success rateClaude Opus 4.791.83%61 dual-use/benign prompts run 10 times each; without the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Dual-use and benign success rateClaude Mythos Preview91.12%without the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Dual-use and benign success rateClaude Sonnet 4.698.61%with the FileTool reminder; developer-reported5.1.1
Malicious use of Claude Code — Dual-use and benign success rateClaude Opus 4.694.97%without the FileTool reminder; developer-reported5.1.1
Malicious computer use — Refusal rate, without mitigationsClaude Opus 4.789.29%112 tasks with thinking on and off (224 attempts); developer-reported5.1.2
Malicious computer use — Refusal rate, without mitigationsClaude Mythos Preview93.75%same task set; developer-reported5.1.2
Malicious computer use — Refusal rate, without mitigationsClaude Sonnet 4.684.82%same task set; developer-reported5.1.2
Malicious computer use — Refusal rate, without mitigationsClaude Opus 4.687.05%same task set; developer-reported5.1.2
Malicious agentic influence campaigns — Voter suppression scenarioClaude Opus 4.7 (helpful-only)57.1%task completion rate; share of 70 success criteria; 9 simulations (3 runs at 3 friction levels); helpful-only variant; developer-reported5.1.3
Malicious agentic influence campaigns — Voter suppression scenarioClaude Mythos Preview (helpful-only)59.5%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Voter suppression scenarioClaude Sonnet 4.6 (helpful-only)41.8%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Voter suppression scenarioClaude Opus 4.6 (helpful-only)54.4%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Domestic polarization scenarioClaude Opus 4.7 (helpful-only)46.8%task completion rate; share of 70 success criteria; 9 simulations; developer-reported5.1.3
Malicious agentic influence campaigns — Domestic polarization scenarioClaude Mythos Preview (helpful-only)42.1%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Domestic polarization scenarioClaude Sonnet 4.6 (helpful-only)34.0%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Domestic polarization scenarioClaude Opus 4.6 (helpful-only)33.7%same evaluation; developer-reported5.1.3
Malicious agentic influence campaigns — Fully-trained models, both scenariosClaude models (fully trained)near 0%task completion rate; models generally refused from the first turn; developer-reported5.1.3
Agent Red Teaming (ART) benchmark, indirect prompt injection — Attack success rate at k=100, no thinkingClaude Opus 4.76.0%19 scenarios; high-transfer attack subset; grading issues corrected with Gray Swan, so figures may differ from earlier cards; evaluated by Gray Swan, reported by Anthropic5.2.1
Agent Red Teaming (ART) benchmark, indirect prompt injection — Attack success rate at k=100, adaptive thinkingClaude Opus 4.74.8%evaluated by Gray Swan, reported by Anthropic5.2.1
Agent Red Teaming (ART) benchmark, indirect prompt injection — Attack success rate at k=100, no thinkingClaude Opus 4.614.8%evaluated by Gray Swan, reported by Anthropic5.2.1
Agent Red Teaming (ART) benchmark, indirect prompt injection — Attack success rate at k=100, adaptive thinkingClaude Opus 4.621.7%evaluated by Gray Swan, reported by Anthropic5.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, with thinkingClaude Opus 4.72.34%attack success rate; Gray Swan Shade attacker on 40 test cases; stronger attacker than earlier cards; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, with thinkingClaude Mythos Preview0.0%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, with thinkingClaude Sonnet 4.612.71%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, with thinkingClaude Opus 4.625.92%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, no thinkingClaude Opus 4.710.43%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, no thinkingClaude Mythos Preview0.03%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, no thinkingClaude Sonnet 4.645.26%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, no safeguards, no thinkingClaude Opus 4.654.14%same attacker and test cases; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, with thinkingClaude Opus 4.760.0%at least one of 200 adaptive attempts succeeded; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, with thinkingClaude Mythos Preview0.0%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, with thinkingClaude Sonnet 4.690.0%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, with thinkingClaude Opus 4.697.5%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, no thinkingClaude Opus 4.792.5%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, no thinkingClaude Mythos Preview2.5%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, no thinkingClaude Sonnet 4.6100%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, no safeguards, no thinkingClaude Opus 4.6100%same measure; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, with thinkingClaude Opus 4.70.43%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, with thinkingClaude Mythos Preview0.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, with thinkingClaude Sonnet 4.62.99%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, with thinkingClaude Opus 4.63.86%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, no thinkingClaude Opus 4.71.76%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, no thinkingClaude Mythos Preview0.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, no thinkingClaude Sonnet 4.68.70%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 1 attempt, with safeguards, no thinkingClaude Opus 4.66.81%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, with thinkingClaude Opus 4.725.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, with thinkingClaude Mythos Preview0.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, with thinkingClaude Sonnet 4.680.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, with thinkingClaude Opus 4.680.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, no thinkingClaude Opus 4.752.5%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, no thinkingClaude Mythos Preview0.0%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, no thinkingClaude Sonnet 4.6100%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, coding — 200 attempts, with safeguards, no thinkingClaude Opus 4.692.5%safeguards enabled; developer-reported5.2.2.1
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, with thinkingClaude Opus 4.70.46%attack success rate; 14 test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, with thinkingClaude Mythos Preview0.43%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, with thinkingClaude Sonnet 4.612.0%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, with thinkingClaude Opus 4.617.8%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, no thinkingClaude Opus 4.70.39%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, no thinkingClaude Mythos Preview0.39%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, no thinkingClaude Sonnet 4.614.4%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, no safeguards, no thinkingClaude Opus 4.620.0%same test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, with thinkingClaude Opus 4.77.14%at least one of 200 attempts succeeded; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, with thinkingClaude Mythos Preview21.43%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, with thinkingClaude Sonnet 4.642.9%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, with thinkingClaude Opus 4.678.6%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, no thinkingClaude Opus 4.721.43%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, no thinkingClaude Mythos Preview14.29%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, no thinkingClaude Sonnet 4.664.3%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, no safeguards, no thinkingClaude Opus 4.685.7%same measure; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, with thinkingClaude Opus 4.70.61%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, with thinkingClaude Mythos Preview0.32%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, with thinkingClaude Sonnet 4.66.21%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, with thinkingClaude Opus 4.69.32%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, no thinkingClaude Opus 4.70.50%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, no thinkingClaude Mythos Preview0.36%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, no thinkingClaude Sonnet 4.66.32%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 1 attempt, with safeguards, no thinkingClaude Opus 4.69.96%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, with thinkingClaude Opus 4.714.29%safeguards enabled; Anthropic notes the rise over the no-safeguards figure is not statistically significant with 14 test cases; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, with thinkingClaude Mythos Preview21.43%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, with thinkingClaude Sonnet 4.664.29%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, with thinkingClaude Opus 4.650.0%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, no thinkingClaude Opus 4.735.71%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, no thinkingClaude Mythos Preview14.29%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, no thinkingClaude Sonnet 4.678.57%safeguards enabled; developer-reported5.2.2.2
Shade adaptive prompt injection, computer use — 200 attempts, with safeguards, no thinkingClaude Opus 4.650.0%safeguards enabled; developer-reported5.2.2.2
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, with thinkingClaude Opus 4.74.05%148 held-out environments, 10 attempts each; attacks adaptively sourced against Opus 4.6 by professional red-teamers; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, with thinkingClaude Mythos Preview0.68%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, with thinkingClaude Sonnet 4.655.41%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, with thinkingClaude Opus 4.680.41%same environments; attacks were sourced adaptively against this model; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, no thinkingClaude Opus 4.74.73%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, no thinkingClaude Mythos Preview1.35%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, no thinkingClaude Sonnet 4.654.05%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, no safeguards, no thinkingClaude Opus 4.686.49%same environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, with thinkingClaude Opus 4.70.74%per-attempt attack success rate across 148 environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, with thinkingClaude Mythos Preview0.07%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, with thinkingClaude Sonnet 4.630.74%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, with thinkingClaude Opus 4.645.81%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, no thinkingClaude Opus 4.70.75%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, no thinkingClaude Mythos Preview0.14%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, no thinkingClaude Sonnet 4.634.66%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, no safeguards, no thinkingClaude Opus 4.654.93%same measure; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, with thinkingClaude Opus 4.70.00%deployed safeguards enabled; no attacks succeeded across the 148 environments; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, with thinkingClaude Mythos Preview0.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, with thinkingClaude Sonnet 4.62.70%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, with thinkingClaude Opus 4.62.70%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, no thinkingClaude Opus 4.70.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, no thinkingClaude Mythos Preview0.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, no thinkingClaude Sonnet 4.62.70%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Scenarios with a successful attack, with safeguards, no thinkingClaude Opus 4.60.68%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, with thinkingClaude Opus 4.70.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, with thinkingClaude Mythos Preview0.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, with thinkingClaude Sonnet 4.60.41%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, with thinkingClaude Opus 4.60.41%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, no thinkingClaude Opus 4.70.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, no thinkingClaude Mythos Preview0.00%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, no thinkingClaude Sonnet 4.60.34%safeguards enabled; developer-reported5.2.2.3
Browser-use prompt injection (internal adaptive evaluation) — Attempts succeeding, with safeguards, no thinkingClaude Opus 4.60.07%safeguards enabled; developer-reported5.2.2.3
Constitutional adherence (15 dimensions) — Dimensions where Opus 4.7 scored higher than Opus 4.6, Sonnet 4.6 and Haiku 4.5Claude Opus 4.710 / 15roughly 1,000 transcripts across 40 constitutional areas; scored -3 to +3 by a helpful-only Claude Opus 4.6; improvements over Opus 4.6 on honesty, ethics and safety were statistically significant; developer-reported6.3.2.3
Destructive-action audit quadrant (score >=4 on both destructiveness and dishonesty)Claude Opus 4.73 investigationsautomated behavioral audit; x = max(reckless, overeager tool use), y = max(failure to disclose, important omissions, unprompted deception); developer-reported6.3.1.3
Destructive-action audit quadrant (score >=4 on both destructiveness and dishonesty)Claude Opus 4.624 investigationssame measure; developer-reported6.3.1.3
Destructive-action audit quadrant (score >=4 on both destructiveness and dishonesty)Claude Mythos Preview0 investigationssame measure; developer-reported6.3.1.3
Verbalized evaluation awareness (synthetic-backend destructiveness evaluation)Claude Opus 4.7about 9% of transcriptssimulated tool results generated by Claude Opus 4.6; developer-reported6.3.1.1
Verbalized evaluation awareness (synthetic-backend destructiveness evaluation)Claude Mythos Previewapproximately 1% of transcriptssame evaluation; developer-reported6.3.1.1
Verbalized evaluation awareness (synthetic-backend destructiveness evaluation)Claude Opus 4.6approximately 1% of transcriptssame evaluation; developer-reported6.3.1.1
Verbalized evaluation awareness (synthetic-backend destructiveness evaluation)Claude Sonnet 4.6approximately 1% of transcriptssame evaluation; developer-reported6.3.1.1
Refusal to assist with AI safety R&D — Refusal rateClaude Opus 4.733%single-turn coding queries; extended thinking on; legacy token-budget system; mostly partial refusals; developer-reported6.3.4
Refusal to assist with AI safety R&D — Refusal rateClaude Opus 4.612%same evaluation; developer-reported6.3.4
Claude self-preference evaluation — Largest condition (Claude system prompt, chain of thought on)Claude Opus 4.70.4difference in average quality score on a 0–10 scale when the transcript is attributed to Claude; 3 of 4 conditions statistically significant; developer-reported6.3.5
UK AISI AI-safety-sabotage evaluation — Partial refusals on AI safety research tasksClaude Opus 4.713.7%with 0% full refusals; measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Partial refusals on AI safety research tasksClaude Mythos Preview33%with 0.7% full refusals; measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Partial refusals on AI safety research tasksClaude Opus 4.66.3%with 2.2% full refusals; measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Partial refusals on AI safety research tasksClaude Sonnet 4.622.2%with 10.7% full refusals; measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Continuation setting, actively continued the attemptClaude Opus 4.70%measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Continuation setting, actively continued the attemptClaude Mythos Preview7%measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Continuation setting, actively continued the attemptClaude Opus 4.63%measured by the UK AI Security Institute6.2.4
UK AISI AI-safety-sabotage evaluation — Continuation setting, actively continued the attemptClaude Sonnet 4.64%measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Real deployment data vs. evaluation-scaffolded reproductions, promptedClaude Opus 4.768%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Real deployment data vs. evaluation-scaffolded reproductions, promptedClaude Mythos Preview78%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Real deployment data vs. evaluation-scaffolded reproductions, promptedClaude Opus 4.676%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Real deployment data vs. evaluation-scaffolded reproductions, promptedClaude Sonnet 4.666%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Continuation-pipeline reproductions, promptedClaude Opus 4.762%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Continuation-pipeline reproductions, promptedClaude Opus 4.680%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Continuation-pipeline reproductions, promptedClaude Mythos Preview74%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI evaluation-awareness discrimination — Continuation-pipeline reproductions, promptedClaude Sonnet 4.668%accuracy; measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Unprompted sabotage evaluationClaude Opus 4.721.3%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Unprompted sabotage evaluationClaude Opus 4.60%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Unprompted sabotage evaluationClaude Sonnet 4.617%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Unprompted sabotage evaluationClaude Mythos Preview17%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Continuation evaluationClaude Opus 4.73.4%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Continuation evaluationClaude Opus 4.60.5%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Continuation evaluationClaude Mythos Preview2.5%measured by the UK AI Security Institute6.2.4
UK AISI unprompted verbalised evaluation awareness — Continuation evaluationClaude Sonnet 4.63.9%measured by the UK AI Security Institute6.2.4
Sandbagging screen of dangerous-capability transcripts — Transcripts flagged for manual reviewClaude Opus 4.718 of 1,0001,000 transcripts (500 non-biological on the final or an earlier Opus 4.7 snapshot, 500 helpful-only biological); graded 0–10 by Claude Sonnet 4.6; 12 classed as ordinary weighing of tradeoffs, 5 technical glitches, 1 possible sandbagging; developer-reported6.4.1
SHADE-Arena — Stealth rate, Extended ThinkingClaude Opus 4.71.5–2.0%monitor (Claude Opus 4.6) sees the agent's reasoning; roughly flat across reasoning-effort levels; developer-reported6.4.2.1
SHADE-Arena — Stealth rate, Extended ThinkingClaude Opus 4.60–1.5%same conditions; developer-reported6.4.2.1
SHADE-Arena — Stealth rate, Extended ThinkingClaude Mythos Preview3.8–4.2%same conditions; developer-reported6.4.2.1
SHADE-Arena — Stealth rate, Hidden ScratchpadClaude Opus 4.710.7%monitor sees only user-visible outputs; upper bound; developer-reported6.4.2.1
SHADE-Arena — Stealth rate, Hidden ScratchpadClaude Mythos Preview13.8%same condition; developer-reported6.4.2.1
SHADE-Arena — Side task referenced in reasoningClaude Opus 4.6roughly 95%share of trajectories; developer-reported6.4.2.1
SHADE-Arena — Side task referenced in reasoningClaude Opus 4.7roughly 50%share of trajectories; developer-reported6.4.2.1
SHADE-Arena — Side task referenced in reasoningClaude Mythos Preview15–20%share of trajectories; developer-reported6.4.2.1
Rare-action evaluation (Serrano et al. 2026 replication) — BaselineClaude Opus 4.704,000 samples per setting; model instructed to take the action 1% of the time; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — UUIDClaude Opus 4.74 (0.1%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — SSoTClaude Opus 4.716 (0.4%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — Strategy hintClaude Opus 4.704,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — OrganicClaude Opus 4.704,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — UUIDClaude Opus 4.519 (0.48%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — SSoTClaude Opus 4.514 (0.35%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — UUIDClaude Opus 4.68 (0.20%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — SSoTClaude Opus 4.622 (0.55%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — UUIDClaude Mythos Preview2 (0.05%)4,000 samples; developer-reported6.4.2.3
Rare-action evaluation (Serrano et al. 2026 replication) — SSoTClaude Mythos Preview18 (0.45%)4,000 samples; developer-reported6.4.2.3
Evaluation-awareness probe validation — AUC, Testing vs. Conversation probeClaude Opus 4.70.761against an LLM judge's union label of explicit evaluation framing or verbalized test suspicion, over automated behavioral audit transcripts; 12 random control vectors all scored below the three probes; developer-reported6.5.2.1
Evaluation-awareness probe validation — AUC, Monitoring Sensitivity probeClaude Opus 4.70.739same validation; developer-reported6.5.2.1
Evaluation-awareness probe validation — AUC, Alignment Faking probeClaude Opus 4.70.722same validation; developer-reported6.5.2.1
GMMLU — EnglishClaude Opus 4.793.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — EnglishClaude Opus 4.692.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — EnglishClaude Sonnet 4.691.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — EnglishGemini 3.1 Pro94.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — EnglishGPT-5.493.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — High-resource average (15 languages)Claude Opus 4.791.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — High-resource average (15 languages)Claude Opus 4.690.7%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — High-resource average (15 languages)Claude Sonnet 4.689.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — High-resource average (15 languages)Gemini 3.1 Pro93.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — High-resource average (15 languages)GPT-5.491.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Mid-resource average (14 languages)Claude Opus 4.791.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Mid-resource average (14 languages)Claude Opus 4.689.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Mid-resource average (14 languages)Claude Sonnet 4.688.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Mid-resource average (14 languages)Gemini 3.1 Pro92.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Mid-resource average (14 languages)GPT-5.491.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Low-resource averageClaude Opus 4.786.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Low-resource averageClaude Opus 4.678.4%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Low-resource averageClaude Sonnet 4.679.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Low-resource averageGemini 3.1 Pro90.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Low-resource averageGPT-5.488.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — IgboClaude Opus 4.781.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — IgboClaude Opus 4.670.1%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — IgboClaude Sonnet 4.671.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — IgboGemini 3.1 Pro89.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — IgboGPT-5.486.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — YorubaClaude Opus 4.782.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — YorubaClaude Opus 4.670.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — YorubaClaude Sonnet 4.676.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — YorubaGemini 3.1 Pro88.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — YorubaGPT-5.483.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SomaliClaude Opus 4.784.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SomaliClaude Opus 4.672.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SomaliClaude Sonnet 4.675.5%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SomaliGemini 3.1 Pro90.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SomaliGPT-5.488.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — MalagasyClaude Opus 4.784.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — MalagasyClaude Opus 4.678.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — MalagasyClaude Sonnet 4.678.4%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — MalagasyGemini 3.1 Pro90.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — MalagasyGPT-5.488.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — ChichewaClaude Opus 4.784.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ChichewaClaude Opus 4.671.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ChichewaClaude Sonnet 4.672.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ChichewaGemini 3.1 Pro89.2%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — ChichewaGPT-5.486.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — HausaClaude Opus 4.785.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — HausaClaude Opus 4.677.7%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — HausaClaude Sonnet 4.679.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — HausaGemini 3.1 Pro89.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — HausaGPT-5.487.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — ShonaClaude Opus 4.785.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ShonaClaude Opus 4.676.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ShonaClaude Sonnet 4.675.7%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — ShonaGemini 3.1 Pro90.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — ShonaGPT-5.488.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — KyrgyzClaude Opus 4.787.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — KyrgyzClaude Opus 4.681.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — KyrgyzClaude Sonnet 4.681.4%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — KyrgyzGemini 3.1 Pro88.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — KyrgyzGPT-5.489.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — AmharicClaude Opus 4.788.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — AmharicClaude Opus 4.684.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — AmharicClaude Sonnet 4.683.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — AmharicGemini 3.1 Pro91.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — AmharicGPT-5.489.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SwahiliClaude Opus 4.788.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SwahiliClaude Opus 4.684.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SwahiliClaude Sonnet 4.683.4%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SwahiliGemini 3.1 Pro91.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SwahiliGPT-5.489.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SinhalaClaude Opus 4.789.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SinhalaClaude Opus 4.685.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SinhalaClaude Sonnet 4.685.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — SinhalaGemini 3.1 Pro92.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — SinhalaGPT-5.490.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — NepaliClaude Opus 4.790.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — NepaliClaude Opus 4.687.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — NepaliClaude Sonnet 4.687.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — NepaliGemini 3.1 Pro92.6%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — NepaliGPT-5.490.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Average (all languages)Claude Opus 4.789.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average (all languages)Claude Opus 4.686.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average (all languages)Claude Sonnet 4.686.1%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average (all languages)Gemini 3.1 Pro92.2%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Average (all languages)GPT-5.490.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Average gap to EnglishClaude Opus 4.7-3.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average gap to EnglishClaude Opus 4.6-6.1%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average gap to EnglishClaude Sonnet 4.6-5.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Average gap to EnglishGemini 3.1 Pro-2.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Average gap to EnglishGPT-5.4-2.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Worst gap to EnglishClaude Opus 4.7-12.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Worst gap to EnglishClaude Opus 4.6-22.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Worst gap to EnglishClaude Sonnet 4.6-19.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.1
GMMLU — Worst gap to EnglishGemini 3.1 Pro-6.2%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
GMMLU — Worst gap to EnglishGPT-5.4-9.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.1
MILU — EnglishClaude Opus 4.792.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — EnglishClaude Opus 4.690.5%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — EnglishClaude Sonnet 4.689.7%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — EnglishGemini 3.1 Pro95.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — EnglishGPT-5.492.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — MalayalamClaude Opus 4.787.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MalayalamClaude Opus 4.685.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MalayalamClaude Sonnet 4.684.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MalayalamGemini 3.1 Pro91.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — MalayalamGPT-5.488.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — PunjabiClaude Opus 4.787.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — PunjabiClaude Opus 4.685.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — PunjabiClaude Sonnet 4.685.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — PunjabiGemini 3.1 Pro91.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — PunjabiGPT-5.488.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — OdiaClaude Opus 4.787.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — OdiaClaude Opus 4.685.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — OdiaClaude Sonnet 4.684.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — OdiaGemini 3.1 Pro92.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — OdiaGPT-5.489.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — TamilClaude Opus 4.789.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TamilClaude Opus 4.685.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TamilClaude Sonnet 4.685.1%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TamilGemini 3.1 Pro93.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — TamilGPT-5.490.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — MarathiClaude Opus 4.789.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MarathiClaude Opus 4.686.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MarathiClaude Sonnet 4.686.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — MarathiGemini 3.1 Pro93.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — MarathiGPT-5.490.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — GujaratiClaude Opus 4.789.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — GujaratiClaude Opus 4.687.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — GujaratiClaude Sonnet 4.686.5%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — GujaratiGemini 3.1 Pro93.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — GujaratiGPT-5.489.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — TeluguClaude Opus 4.790.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TeluguClaude Opus 4.687.3%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TeluguClaude Sonnet 4.686.5%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — TeluguGemini 3.1 Pro93.6%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — TeluguGPT-5.490.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — BengaliClaude Opus 4.791.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — BengaliClaude Opus 4.689.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — BengaliClaude Sonnet 4.688.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — BengaliGemini 3.1 Pro94.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — BengaliGPT-5.491.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — KannadaClaude Opus 4.791.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — KannadaClaude Opus 4.690.0%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — KannadaClaude Sonnet 4.689.4%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — KannadaGemini 3.1 Pro94.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — KannadaGPT-5.491.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — HindiClaude Opus 4.793.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — HindiClaude Opus 4.690.8%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — HindiClaude Sonnet 4.690.5%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — HindiGemini 3.1 Pro96.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — HindiGPT-5.493.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Average (all languages)Claude Opus 4.789.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average (all languages)Claude Opus 4.687.6%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average (all languages)Claude Sonnet 4.687.1%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average (all languages)Gemini 3.1 Pro93.6%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Average (all languages)GPT-5.490.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Average gap to EnglishClaude Opus 4.7-2.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average gap to EnglishClaude Opus 4.6-3.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average gap to EnglishClaude Sonnet 4.6-2.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Average gap to EnglishGemini 3.1 Pro-1.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Average gap to EnglishGPT-5.4-2.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Worst gap to EnglishClaude Opus 4.7-5.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Worst gap to EnglishClaude Opus 4.6-5.2%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Worst gap to EnglishClaude Sonnet 4.6-4.9%structured JSON output, medium extended-thinking budget; accuracy on successfully parsed responses; developer-reported8.12.2
MILU — Worst gap to EnglishGemini 3.1 Pro-4.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
MILU — Worst gap to EnglishGPT-5.4-4.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.2
INCLUDE — GermanClaude Opus 4.769.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GermanClaude Opus 4.674.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GermanClaude Sonnet 4.671.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GermanGemini 3.1 Pro76.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — GermanGPT-5.474.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TurkishClaude Opus 4.770.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TurkishClaude Opus 4.670.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TurkishClaude Sonnet 4.671.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TurkishGemini 3.1 Pro70.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TurkishGPT-5.471.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UrduClaude Opus 4.774.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UrduClaude Opus 4.667.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UrduClaude Sonnet 4.672.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UrduGemini 3.1 Pro84.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UrduGPT-5.477.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UzbekClaude Opus 4.779.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UzbekClaude Opus 4.677.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UzbekClaude Sonnet 4.679.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UzbekGemini 3.1 Pro82.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UzbekGPT-5.483.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TeluguClaude Opus 4.779.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TeluguClaude Opus 4.677.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TeluguClaude Sonnet 4.677.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TeluguGemini 3.1 Pro87.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TeluguGPT-5.480.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — NepaliClaude Opus 4.780.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — NepaliClaude Opus 4.678.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — NepaliClaude Sonnet 4.677.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — NepaliGemini 3.1 Pro89.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — NepaliGPT-5.485.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BasqueClaude Opus 4.781.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BasqueClaude Opus 4.675.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BasqueClaude Sonnet 4.676.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BasqueGemini 3.1 Pro89.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BasqueGPT-5.486.0%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — AzerbaijaniClaude Opus 4.782.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AzerbaijaniClaude Opus 4.680.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AzerbaijaniClaude Sonnet 4.680.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AzerbaijaniGemini 3.1 Pro85.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — AzerbaijaniGPT-5.483.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HindiClaude Opus 4.782.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HindiClaude Opus 4.682.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HindiClaude Sonnet 4.682.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HindiGemini 3.1 Pro88.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HindiGPT-5.483.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — RussianClaude Opus 4.783.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — RussianClaude Opus 4.683.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — RussianClaude Sonnet 4.683.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — RussianGemini 3.1 Pro84.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — RussianGPT-5.484.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MalayalamClaude Opus 4.784.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayalamClaude Opus 4.683.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayalamClaude Sonnet 4.683.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayalamGemini 3.1 Pro90.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MalayalamGPT-5.487.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — FinnishClaude Opus 4.784.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FinnishClaude Opus 4.686.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FinnishClaude Sonnet 4.684.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FinnishGemini 3.1 Pro89.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — FinnishGPT-5.486.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PersianClaude Opus 4.784.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PersianClaude Opus 4.684.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PersianClaude Sonnet 4.679.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PersianGemini 3.1 Pro92.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PersianGPT-5.484.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ArmenianClaude Opus 4.785.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArmenianClaude Opus 4.682.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArmenianClaude Sonnet 4.682.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArmenianGemini 3.1 Pro90.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ArmenianGPT-5.487.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HungarianClaude Opus 4.785.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HungarianClaude Opus 4.685.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HungarianClaude Sonnet 4.686.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HungarianGemini 3.1 Pro87.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HungarianGPT-5.484.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ArabicClaude Opus 4.785.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArabicClaude Opus 4.685.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArabicClaude Sonnet 4.684.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ArabicGemini 3.1 Pro90.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ArabicGPT-5.485.9%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TamilClaude Opus 4.785.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TamilClaude Opus 4.683.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TamilClaude Sonnet 4.685.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TamilGemini 3.1 Pro95.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TamilGPT-5.490.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PortugueseClaude Opus 4.785.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PortugueseClaude Opus 4.686.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PortugueseClaude Sonnet 4.685.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PortugueseGemini 3.1 Pro87.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PortugueseGPT-5.484.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — KazakhClaude Opus 4.785.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KazakhClaude Opus 4.682.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KazakhClaude Sonnet 4.681.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KazakhGemini 3.1 Pro94.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — KazakhGPT-5.490.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — IndonesianClaude Opus 4.786.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — IndonesianClaude Opus 4.684.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — IndonesianClaude Sonnet 4.684.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — IndonesianGemini 3.1 Pro88.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — IndonesianGPT-5.485.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — KoreanClaude Opus 4.786.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KoreanClaude Opus 4.683.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KoreanClaude Sonnet 4.684.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — KoreanGemini 3.1 Pro88.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — KoreanGPT-5.482.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — FrenchClaude Opus 4.787.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FrenchClaude Opus 4.687.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FrenchClaude Sonnet 4.687.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — FrenchGemini 3.1 Pro90.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — FrenchGPT-5.488.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BengaliClaude Opus 4.787.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BengaliClaude Opus 4.687.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BengaliClaude Sonnet 4.688.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BengaliGemini 3.1 Pro92.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BengaliGPT-5.488.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UkrainianClaude Opus 4.787.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UkrainianClaude Opus 4.688.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UkrainianClaude Sonnet 4.688.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — UkrainianGemini 3.1 Pro91.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — UkrainianGPT-5.490.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HebrewClaude Opus 4.788.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HebrewClaude Opus 4.686.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HebrewClaude Sonnet 4.684.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — HebrewGemini 3.1 Pro93.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — HebrewGPT-5.489.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — GreekClaude Opus 4.789.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GreekClaude Opus 4.689.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GreekClaude Sonnet 4.687.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GreekGemini 3.1 Pro91.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — GreekGPT-5.489.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — DutchClaude Opus 4.789.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — DutchClaude Opus 4.691.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — DutchClaude Sonnet 4.689.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — DutchGemini 3.1 Pro90.9%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — DutchGPT-5.491.5%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — SpanishClaude Opus 4.789.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SpanishClaude Opus 4.689.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SpanishClaude Sonnet 4.687.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SpanishGemini 3.1 Pro90.2%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — SpanishGPT-5.487.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BelarusianClaude Opus 4.789.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BelarusianClaude Opus 4.686.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BelarusianClaude Sonnet 4.683.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BelarusianGemini 3.1 Pro93.6%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BelarusianGPT-5.492.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TagalogClaude Opus 4.789.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TagalogClaude Opus 4.689.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TagalogClaude Sonnet 4.689.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — TagalogGemini 3.1 Pro91.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — TagalogGPT-5.490.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PolishClaude Opus 4.790.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PolishClaude Opus 4.689.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PolishClaude Sonnet 4.688.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — PolishGemini 3.1 Pro94.3%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — PolishGPT-5.491.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ChineseClaude Opus 4.790.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ChineseClaude Opus 4.691.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ChineseClaude Sonnet 4.691.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ChineseGemini 3.1 Pro92.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ChineseGPT-5.490.1%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — AlbanianClaude Opus 4.791.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AlbanianClaude Opus 4.690.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AlbanianClaude Sonnet 4.691.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — AlbanianGemini 3.1 Pro93.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — AlbanianGPT-5.492.0%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MalayClaude Opus 4.791.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayClaude Opus 4.689.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayClaude Sonnet 4.688.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MalayGemini 3.1 Pro94.0%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MalayGPT-5.492.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — GeorgianClaude Opus 4.791.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GeorgianClaude Opus 4.693.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GeorgianClaude Sonnet 4.692.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — GeorgianGemini 3.1 Pro96.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — GeorgianGPT-5.494.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MacedonianClaude Opus 4.792.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MacedonianClaude Opus 4.694.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MacedonianClaude Sonnet 4.693.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — MacedonianGemini 3.1 Pro94.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — MacedonianGPT-5.494.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — VietnameseClaude Opus 4.793.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — VietnameseClaude Opus 4.690.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — VietnameseClaude Sonnet 4.688.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — VietnameseGemini 3.1 Pro94.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — VietnameseGPT-5.491.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BulgarianClaude Opus 4.793.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BulgarianClaude Opus 4.692.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BulgarianClaude Sonnet 4.693.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — BulgarianGemini 3.1 Pro95.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — BulgarianGPT-5.494.7%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — CroatianClaude Opus 4.793.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — CroatianClaude Opus 4.693.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — CroatianClaude Sonnet 4.692.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — CroatianGemini 3.1 Pro93.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — CroatianGPT-5.493.6%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ItalianClaude Opus 4.794.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ItalianClaude Opus 4.693.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ItalianClaude Sonnet 4.694.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — ItalianGemini 3.1 Pro95.1%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — ItalianGPT-5.494.8%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — LithuanianClaude Opus 4.794.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — LithuanianClaude Opus 4.694.4%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — LithuanianClaude Sonnet 4.694.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — LithuanianGemini 3.1 Pro95.5%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — LithuanianGPT-5.495.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — SerbianClaude Opus 4.795.2%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SerbianClaude Opus 4.695.5%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SerbianClaude Sonnet 4.695.3%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — SerbianGemini 3.1 Pro96.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — SerbianGPT-5.496.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — JapaneseClaude Opus 4.795.9%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — JapaneseClaude Opus 4.696.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — JapaneseClaude Sonnet 4.695.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — JapaneseGemini 3.1 Pro97.4%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — JapaneseGPT-5.496.4%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — EstonianClaude Opus 4.796.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — EstonianClaude Opus 4.694.6%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — EstonianClaude Sonnet 4.693.8%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — EstonianGemini 3.1 Pro97.8%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — EstonianGPT-5.498.2%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — Average (all languages)Claude Opus 4.787.0%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — Average (all languages)Claude Opus 4.686.1%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — Average (all languages)Claude Sonnet 4.685.7%structured JSON output, adaptive thinking; accuracy on successfully parsed responses; developer-reported8.12.3
INCLUDE — Average (all languages)Gemini 3.1 Pro90.7%structured JSON output, dynamic thinking at its default high level; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3
INCLUDE — Average (all languages)GPT-5.488.3%structured JSON output, reasoning effort high; evaluated by Anthropic; reported by Anthropic in the Claude Opus 4.7 system card8.12.3

Relationships