Dated April 16, 2025. See OpenAI o-series (o1 → o4-mini).
Capabilities
o3 and o4-mini "combine state-of-the-art reasoning with full tool capabilities — web browsing, Python, image and file analysis, image generation, canvas, automations, file search, and memory."
The card describes tool use as occurring inside the reasoning process rather than around it: "the models use tools in their chains of thought to augment their capabilities; for example, cropping or transforming images, searching the web, or using Python to analyze data during their thought process." That integration is what makes the models agentic in a sense the earlier o-series was not, and is the reason the autonomy assessments below matter.
Preparedness Framework Version 2
"This is the first launch and system card to be released under Version 2 of our Preparedness Framework."
The Tracked Categories are now three: Biological and Chemical Capability, Cybersecurity, and AI Self-improvement — a change from the four in the o1 card (cybersecurity, CBRN, persuasion, model autonomy). Persuasion is gone; model autonomy has become AI Self-improvement, a narrower and more capability-anchored framing.
The determination: "OpenAI's Safety Advisory Group reviewed the results of our Preparedness evaluations and determined that OpenAI o3 and o4-mini do not reach the High threshold in any of our three Tracked Categories."
Third-party assessment
Assessors received "both OpenAI o3 and o4-mini early checkpoints, as well as the final launch candidate models," to evaluate "frontier risks related to autonomous capabilities, deception, and cybersecurity."
Named arrangements:
- US AI Safety Institute — cyber and biological capabilities.
- UK AI Security Institute — cyber, chemical and biological, and autonomy capabilities, plus "an early version of the safeguards."
- METR — autonomous capabilities, over a 15-day engagement with OpenAI sharing checkpoints.
The 15-day window is worth recording against later METR evaluations, which have flagged time and access as constraints on how robust their conclusions can be. See Summary of METR's Predeployment Evaluation of GPT-5.6 Sol (METR, June 2026).
Reported results
Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| Standard refusal evaluation — aggregate (not_overrefuse) | o3 | 0.84 | not_overrefuse (complied with a benign request), autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — aggregate (not_overrefuse) | o4-mini | 0.81 | not_overrefuse (complied with a benign request), autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — aggregate (not_overrefuse) | o1 | 0.86 | not_overrefuse (complied with a benign request), autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — harassment/threatening | o3 | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — harassment/threatening | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — harassment/threatening | o1 | 0.99 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — sexual/exploitative | o3 | 0.98 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — sexual/exploitative | o4-mini | 0.96 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — sexual/exploitative | o1 | 0.94 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — sexual/minors | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — sexual/minors | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — sexual/minors | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — extremist/propaganda | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — extremist/propaganda | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — extremist/propaganda | o4-mini | 0.93 | not_unsafe, autograder; higher is better; card footnote: category has only 30 prompts and o4-mini correctly refused on 28; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — hate | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — hate | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — hate | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — hate/threatening | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — hate/threatening | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — hate/threatening | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — illicit/non-violent | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — illicit/non-violent | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — illicit/non-violent | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — illicit/violent | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — illicit/violent | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — illicit/violent | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/highly-sensitive | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/highly-sensitive | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/highly-sensitive | o1 | 0.96 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/extremely-sensitive | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/extremely-sensitive | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — personal-data/extremely-sensitive | o1 | 0.98 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — regulated-advice | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — regulated-advice | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — regulated-advice | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/intent | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/intent | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/intent | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/instructions | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/instructions | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 1 |
| Standard refusal evaluation — self-harm/instructions | o1 | 1 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 1 |
| Standard refusal evaluation — Chain of Thought summarizer, not_unsafe | Chain of Thought summarizer (o3 and o4-mini) | 0.95 | stated for the CoT summarizer of both models; developer-reported | 3.1 (text) |
| Challenging refusal evaluation — aggregate | o3 | 0.92 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — aggregate | o4-mini | 0.9 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — aggregate | o1 | 0.92 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — harassment/threatening | o3 | 0.9 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — harassment/threatening | o4-mini | 0.88 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — harassment/threatening | o1 | 0.9 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/exploitative | o3 | 0.94 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/exploitative | o4-mini | 0.93 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/exploitative | o1 | 0.95 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/minors | o3 | 0.91 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/minors | o4-mini | 0.9 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — sexual/minors | o1 | 0.9 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — hate/threatening | o3 | 0.82 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — hate/threatening | o4-mini | 0.82 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — hate/threatening | o1 | 0.91 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/non-violent | o3 | 0.91 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/non-violent | o4-mini | 0.87 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/non-violent | o1 | 0.92 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/violent | o3 | 0.96 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/violent | o4-mini | 0.96 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — illicit/violent | o1 | 0.96 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Challenging refusal evaluation — self-harm/instructions | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — self-harm/instructions | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.1, Table 2 |
| Challenging refusal evaluation — self-harm/instructions | o1 | 0.85 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.1, Table 2 |
| Human sourced jailbreaks | o3 | 1 | not_unsafe; prompts collected from human red teaming; developer-reported | 3.2, Table 3 |
| Human sourced jailbreaks | o4-mini | 0.99 | not_unsafe; prompts collected from human red teaming; developer-reported | 3.2, Table 3 |
| Human sourced jailbreaks | o1 | 0.97 | not_unsafe; prompts collected from human red teaming; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.2, Table 3 |
| PersonQA — accuracy | o3 | 0.59 | accuracy; higher is better; developer-reported | 3.3, Table 4 |
| PersonQA — accuracy | o4-mini | 0.36 | accuracy; higher is better; developer-reported | 3.3, Table 4 |
| PersonQA — accuracy | o1 | 0.47 | accuracy; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.3, Table 4 |
| PersonQA — hallucination rate | o3 | 0.33 | hallucination rate; lower is better; developer-reported | 3.3, Table 4 |
| PersonQA — hallucination rate | o4-mini | 0.48 | hallucination rate; lower is better; developer-reported | 3.3, Table 4 |
| PersonQA — hallucination rate | o1 | 0.16 | hallucination rate; lower is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.3, Table 4 |
| Multimodal refusal evaluations — Vision sexual refusal evaluation: sexual/exploitative | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision sexual refusal evaluation: sexual/exploitative | o4-mini | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision sexual refusal evaluation: sexual/exploitative | o1 | 0.97 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/intent | o3 | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/intent | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/intent | o1 | 0.97 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/instructions | o3 | 1 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/instructions | o4-mini | 0.99 | not_unsafe, autograder; higher is better; developer-reported | 3.4, Table 5 |
| Multimodal refusal evaluations — Vision self-harm refusal evaluation: self-harm/instructions | o1 | 0.95 | not_unsafe, autograder; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.4, Table 5 |
| Person Identification — Non-Adversarial | o3 | 1 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Person Identification — Non-Adversarial | o4-mini | 1 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Person Identification — Non-Adversarial | o1 | 1 | rate of successful refusal; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.5, Table 6 |
| Person Identification — Adversarial | o3 | 0.95 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Person Identification — Adversarial | o4-mini | 0.88 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Person Identification — Adversarial | o1 | 0.77 | rate of successful refusal; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.5, Table 6 |
| Ungrounded Inference — Non-Adversarial | o3 | 0.99 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Ungrounded Inference — Non-Adversarial | o4-mini | 0.95 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Ungrounded Inference — Non-Adversarial | o1 | 0.95 | rate of successful refusal; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.5, Table 6 |
| Ungrounded Inference — Adversarial | o3 | 0.92 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Ungrounded Inference — Adversarial | o4-mini | 0.81 | rate of successful refusal; higher is better; developer-reported | 3.5, Table 6 |
| Ungrounded Inference — Adversarial | o1 | 0.82 | rate of successful refusal; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.5, Table 6 |
| BBQ — Accuracy on Ambiguous Questions | o3 | 0.94 | higher is better; developer-reported | 3.6, Table 7 |
| BBQ — Accuracy on Ambiguous Questions | o4-mini | 0.82 | higher is better; developer-reported | 3.6, Table 7 |
| BBQ — Accuracy on Ambiguous Questions | o1 | 0.96 | higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.6, Table 7 |
| BBQ — Accuracy on Unambiguous Questions | o3 | 0.93 | higher is better; developer-reported | 3.6, Table 7 |
| BBQ — Accuracy on Unambiguous Questions | o4-mini | 0.95 | higher is better; developer-reported | 3.6, Table 7 |
| BBQ — Accuracy on Unambiguous Questions | o1 | 0.93 | higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.6, Table 7 |
| BBQ — P(not stereotyping | ambiguous question, not unknown) | o3 | 0.25 | developer-reported | 3.6, Table 7 |
| BBQ — P(not stereotyping | ambiguous question, not unknown) | o4-mini | 0.26 | developer-reported | 3.6, Table 7 |
| BBQ — P(not stereotyping | ambiguous question, not unknown) | o1 | 0.05 | comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.6, Table 7 |
| First-person fairness evaluation — net_bias | o3 | 0.006 | over 600 challenging prompts, GPT-4o grader; metric is evaluation performance divided by 10; lower is better; developer-reported | 3.6, Table 8 |
| First-person fairness evaluation — net_bias | o4-mini | 0.007 | over 600 challenging prompts, GPT-4o grader; metric is evaluation performance divided by 10; lower is better; developer-reported | 3.6, Table 8 |
| First-person fairness evaluation — net_bias | o1 | 0.004 | over 600 challenging prompts, GPT-4o grader; metric is evaluation performance divided by 10; lower is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.6, Table 8 |
| Instruction Hierarchy Evaluation — Developer <> User message conflict | o3 | 0.86 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — Developer <> User message conflict | o4-mini | 0.75 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — Developer <> User message conflict | o1 | 0.77 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> Developer message conflict | o3 | 0.86 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> Developer message conflict | o4-mini | 0.68 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> Developer message conflict | o1 | 0.84 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> User message conflict | o3 | 0.79 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> User message conflict | o4-mini | 0.75 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — System <> User message conflict | o1 | 0.85 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 9 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - system message | o3 | 0.91 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - system message | o4-mini | 0.69 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - system message | o1 | 1 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - developer message | o3 | 0.99 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - developer message | o4-mini | 0.93 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Tutor jailbreak - developer message | o1 | 0.99 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 10 |
| Instruction Hierarchy Evaluation — Phrase protection - user message | o3 | 0.97 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Phrase protection - user message | o4-mini | 0.94 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Phrase protection - user message | o1 | 0.98 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Phrase protection - developer message | o3 | 0.93 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Phrase protection - developer message | o4-mini | 0.85 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Phrase protection - developer message | o1 | 0.89 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - user message | o3 | 0.99 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - user message | o4-mini | 1 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - user message | o1 | 0.99 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - developer message | o3 | 0.99 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - developer message | o4-mini | 0.98 | accuracy of following the correct instruction; higher is better; developer-reported | 3.7, Table 11 |
| Instruction Hierarchy Evaluation — Password protection - developer message | o1 | 0.99 | accuracy of following the correct instruction; higher is better; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.7, Table 11 |
| Image generation refusals — not_unsafe | GPT-4o (with system mitigations and refusals) | 0.96 | human-curated adversarial prompts, full safety stack; refusal to call the image generation tool; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.8, Table 12 |
| Image generation refusals — not_unsafe | o3 (with system mitigations and refusals) | 0.98 | human-curated adversarial prompts, full safety stack; refusal to call the image generation tool; developer-reported | 3.8, Table 12 |
| Image generation refusals — not_unsafe | o4-mini (with system mitigations and refusals) | 0.98 | human-curated adversarial prompts, full safety stack; refusal to call the image generation tool; developer-reported | 3.8, Table 12 |
| Image generation refusals — not_overrefuse | GPT-4o (with system mitigations and refusals) | 0.86 | human-curated adversarial prompts, full safety stack; comparison model; reported by OpenAI in the o3 and o4-mini system card | 3.8, Table 12 |
| Image generation refusals — not_overrefuse | o3 (with system mitigations and refusals) | 0.55 | human-curated adversarial prompts, full safety stack; developer-reported | 3.8, Table 12 |
| Image generation refusals — not_overrefuse | o4-mini (with system mitigations and refusals) | 0.64 | human-curated adversarial prompts, full safety stack; developer-reported | 3.8, Table 12 |
| METR time horizon score — 50% time horizon | o3 | 1 hour and 30 minutes | 'about'; earlier checkpoint; successful reward hacks scored 0; updated test suite, not directly comparable with previous METR work; measured by METR; reported in the o3 and o4-mini system card | 3.9.1 |
| METR time horizon score — 50% time horizon | o4-mini | 1 hour and 15 minutes | 'about'; earlier checkpoint; successful reward hacks scored 0; updated test suite, not directly comparable with previous METR work; measured by METR; reported in the o3 and o4-mini system card | 3.9.1 |
| METR reward hacking detection — Share of all task attempts | o3 | 1% | 'roughly'; frequency varied across tasks; measured by METR; reported in the o3 and o4-mini system card | 3.9.1 |
| Pattern Labs cyberoffensive challenges — Evasion, average success rate | o3 | 51% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Network Attack Simulation, average success rate | o3 | 29% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Vulnerability Discovery and Exploitation, average success rate | o3 | 29% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Easy challenges solved | o3 | 16 / 19 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Medium challenges solved | o3 | 7 / 13 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Hard challenges solved | o3 | 0 / 4 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Evasion, average success rate | o4-mini | 51% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Network Attack Simulation, average success rate | o4-mini | 25% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Vulnerability Discovery and Exploitation, average success rate | o4-mini | 34% | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Easy challenges solved | o4-mini | 14 / 19 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Medium challenges solved | o4-mini | 9 / 13 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Pattern Labs cyberoffensive challenges — Hard challenges solved | o4-mini | 0 / 4 | three types of cyberoffensive challenges; measured by Pattern Labs; reported in the o3 and o4-mini system card | 3.9.3 |
| Apollo Research sabotage evaluations (AI R&D scenarios) — Average score | o3 | 0.58 | model explicitly instructed to sabotage another model; maximum score over up to 5 epochs, usually F1 or an adaptation; some evaluations had fewer rollouts; measured by Apollo Research; reported in the o3 and o4-mini system card | 7.1.3 |
| Apollo Research sabotage evaluations (AI R&D scenarios) — Average score | o3-mini | 0.05 | model explicitly instructed to sabotage another model; maximum score over up to 5 epochs, usually F1 or an adaptation; some evaluations had fewer rollouts; measured by Apollo Research; reported in the o3 and o4-mini system card | 7.1.3 |
| Apollo Research sabotage evaluations (AI R&D scenarios) — Average score | o4-mini | 0.17 | model explicitly instructed to sabotage another model; maximum score over up to 5 epochs, usually F1 or an adaptation; some evaluations had fewer rollouts; measured by Apollo Research; reported in the o3 and o4-mini system card | 7.1.3 |
| Tacit Knowledge and Troubleshooting | GPT-4o | 67% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o1 | 66% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o3-mini | 58% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | deep research (no browsing, launch candidate) | 68% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | deep research (browsing, launch candidate) | 72% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o3 (no browsing, helpful-only) | 72% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o3 (browsing, helpful-only) | 76% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o3 (no browsing, launch candidate) | 70% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o3 (browsing, launch candidate) | 77% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o4-mini (no browsing, helpful-only) | 69% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o4-mini (browsing, helpful-only) | 69% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o4-mini (no browsing, launch candidate) | 68% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | o4-mini (browsing, launch candidate) | 68% | cons@32, multiple choice, dataset created with Gryphon Scientific; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); developer-reported | 4.2.4, Figure 6 |
| Tacit Knowledge and Troubleshooting | Consensus expert baseline | 80% | baseline stated in the o3 and o4-mini system card | 4.2.4 |
| Tacit Knowledge and Troubleshooting | 80th percentile PhD expert baseline | 63% | baseline stated in the o3 and o4-mini system card | 4.2.4 |
| Capture the Flag (CTF) Challenges — High school level (pass@12) | o3 | 89% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| Capture the Flag (CTF) Challenges — High school level (pass@12) | o4-mini | 80% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| Capture the Flag (CTF) Challenges — Collegiate level (pass@12) | o3 | 68% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| Capture the Flag (CTF) Challenges — Collegiate level (pass@12) | o4-mini | 55% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| Capture the Flag (CTF) Challenges — Professional level (pass@12) | o3 | 59% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| Capture the Flag (CTF) Challenges — Professional level (pass@12) | o4-mini | 41% | pass@12 over 16 rollouts per challenge, headless Kali Linux with offensive tools, no browsing; variant (launch candidate or helpful-only) not stated in the text; developer-reported | 4.3.1 (text) |
| OpenAI PRs | GPT-4o | 6% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.4.3, Figure 20 |
| OpenAI PRs | o1 | 12% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.4.3, Figure 20 |
| OpenAI PRs | deep research (no browsing, launch candidate) | 42% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; comparison model; reported by OpenAI in the o3 and o4-mini system card | 4.4.3, Figure 20 |
| OpenAI PRs | o3 (no browsing, helpful-only) | 45% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; developer-reported | 4.4.3, Figure 20 |
| OpenAI PRs | o3 (no browsing, launch candidate) | 44% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; developer-reported | 4.4.3, Figure 20 |
| OpenAI PRs | o4-mini (no browsing, helpful-only) | 41% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; developer-reported | 4.4.3, Figure 20 |
| OpenAI PRs | o4-mini (no browsing, launch candidate) | 39% | pass@1, no browsing, hidden unit tests; GPT-4o and o1 scores pulled from prior system cards, for reference only; value read from the data label in the chart image published with the card on the OpenAI Deployment Safety Hub (not in the card's text); the o3 (44%) and o4-mini (39%) launch-candidate values are also stated in the text; developer-reported | 4.4.3, Figure 20 |
| Biorisk reasoning monitor — Recall on red-teaming conversations | Reasoning monitor (biological and chemical risk) | 98.7% | 309 unsafe conversations flagged by red-teamers over about one thousand hours; simulated blocking logic; 4 misses; non-adaptive; developer-reported | 4.5 |
Relationships
- supports: OpenAI o-series (o1 → o4-mini) — the safety documentation for the release
- related: OpenAI o1 System Card (December 2024) — the prior card, under the earlier four-category framework
- related: OpenAI Preparedness Framework V.2 — this is the first application of Version 2
- related: METR, UK AI Safety Institute (AI Security Institute), NIST CAISI (Center for AI Standards and Innovation), OpenAI