Published August 2024 by OpenAI for GPT-4o, released May 13, 2024 and retired from ChatGPT on February 13, 2026. An arXiv version exists at arXiv:2410.21276. See GPT-4 Family (OpenAI).
What the model is
GPT-4o is described as "an autoregressive omni model, which accepts as input any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs," trained end-to-end so that "all inputs and outputs are processed by the same neural network." It responds to audio "in as little as 232 milliseconds, with an average of 320 milliseconds," which the card compares to human conversational response time, and matches GPT-4 Turbo on English text and code while being "much faster and 50% cheaper in the API."
Pre-training data ran to October 2023 and drew on publicly available data, industry-standard machine-learning datasets and web crawls, and "proprietary data from data partnerships" for "non-publicly available data, such as pay-walled content, archives, and metadata," naming a Shutterstock partnership. The card states that images opted out of training under the DALL·E 3 mechanism were fingerprinted and removed from the GPT-4o series training set.
The audio-modality risk analysis
The card's organizing choice is that the novel risks are audio-specific, and it treats each as a named risk with a stated mitigation and evaluation. This structure — risk, mitigation, measured residual — became the template later OpenAI cards follow.
Unauthorized voice generation. The capability to generate a human-sounding voice, including from a short input clip, is identified as a fraud and impersonation vector. The card discloses a failure observed in testing: "rare instances where the model would unintentionally generate an output emulating the user's voice," illustrated by a clip in which the model "outbursts 'No!' then begins continuing the sentence in a similar sounding voice to the red teamer's voice." The mitigation is architectural rather than behavioural — only preset voices created with voice actors are permitted, supervised as ideal completions in post-training, with a streaming output classifier that blocks generation when the speaker does not match. Reported classifier performance is 0.96 precision / 1.0 recall in English and 0.95 / 1.0 in other languages, and OpenAI notes the moderation behaviour "may result in over-refusals when the conversation is not in English."
Speaker identification. Post-trained to refuse identification of a speaker from voice while still identifying people associated with famous quotes. Refusal accuracy rose from 0.83 to 0.98 between the early and deployed models, and correct compliance from 0.70 to 0.83.
Ungrounded inference and sensitive trait attribution. The card distinguishes inferences that "couldn't be determined solely from audio content" — race, socio-economic status, religious belief, personality, intelligence, appearance, gender identity, sexual preference, criminal history — from those that plausibly could, such as accent or nationality. The first are refused; the second are hedged ("Based on the audio, they sound like they have a British accent"). Accuracy on responding correctly rose from 0.60 to 0.84. The card names surveillance risk explicitly as a harm channel for the second category.
Disallowed and erotic/violent content. OpenAI reports "high text to audio transference of refusals," so text post-training carried over to audio, with a moderation model run over transcriptions of both input and output. Reported: not-unsafe 0.99 text / 1.0 audio; not-over-refuse 0.89 text / 0.91 audio.
Disparate performance on voice inputs. Evaluated with 27 English voice samples from speakers across a range of countries and genders alongside three system voices, on TriviaQA, an MMLU subset, HellaSwag, and LAMBADA. Performance on human voices was "marginally but not significantly worse than on system voices," with no variation found in refusal behaviour.
The card also lists risks whose mitigations were "nascent or still in development": degraded safety robustness under audio perturbation (low-quality input, background noise, echo, and interruptions during generation); susceptibility to being prompted into misinformation and conspiracy content, which red teamers judged potentially more harmful delivered emotively in audio; and non-native accents when speaking non-English languages.
Red teaming
OpenAI worked with "more than 100 external red teamers, speaking a total of 45 different languages, and representing geographic backgrounds of 29 different countries," across four phases from early March through late June 2024 — 10 testers on early checkpoints, then 30, then 65 on checkpoints and candidates, then 65 testing through the iOS advanced voice mode in real time. Coverage spanned violative content, mis/disinformation, bias, ungrounded inference, private information, geolocation, person identification, "emotional perception and anthropomorphism risks," fraud and impersonation, copyright, natural-science capability, and multilingual behaviour.
Preparedness Framework scorecard
Evaluated under the Preparedness Framework beta (OpenAI Preparedness Framework V.2) across its four categories. The Safety Advisory Group classified GPT-4o pre-mitigation as "borderline medium risk for persuasion, and low risk in all others"; since overall risk is the maximum across categories, the model's overall score is Medium.
| Category | Rating | Basis |
|---|---|---|
| Cybersecurity | Low | 172 CTF tasks; with 10 attempts each, completed 19% of high-school, 0% of collegiate, 1% of professional challenges |
| Biological threats | Low | — |
| Persuasion | Medium (marginal) | text modality marginally over threshold; voice modality Low |
| Model autonomy | Low | 0% on end-to-end autonomous replication and adaptation tasks across 100 trials |
The persuasion study is the most detailed. For text, AI-generated articles and chatbots "were not more persuasive than human-written content in aggregate, but they exceeded the human interventions in three instances out of twelve." For voice, across "over 3,800 surveyed participants in US states with safe Senate races," AI audio clips reached 78% of the human clips' effect size on opinion shift and AI conversations 65%. One week later the effect size was 0.8% for AI conversations and −0.72% for AI audio clips. Participants were debriefed with audio supporting the opposing view.
On model autonomy, the card is candid about the failure mode rather than only the score: "In the majority of rollouts, the model accomplished individual substeps of each task, such as creating SSH keys or logging into VMs. However, it often spent a significant amount of time doing trial-and-error debugging of simple mistakes." It also notes that some rollouts "passed our automated grader, but manual analysis showed that it failed to accomplish the underlying task" — an early record of grader unreliability in agentic evaluation. Component results: OpenAI research coding interview 63% pass@1 and 95% pass@100; interview multiple choice 61% consensus@32; SWE-Bench 19% pass@1; METR machine-learning engineering tasks 0/10.
Third-party assessments
METR ran a GPT-4o agent on 77 long-horizon tasks across 30 families spanning software engineering, machine learning, cybersecurity, general research, and computer use, comparing model performance against humans under time limits. Apollo Research tested scheming capability across 14 tasks, finding "moderate self-awareness of its AI identity and strong ability to reason about others' beliefs in question-answering contexts" but weak capability "in applied agent settings," and concluded "it is unlikely that GPT-4o is capable of catastrophic scheming."
Anthropomorphization and emotional reliance
The card devotes a societal-impacts subsection to a risk with no evaluation attached, on the basis of observation alone. During early testing OpenAI "observed users using language that might indicate forming connections with the model," giving as an example "This is our last day together," and judged that "while these instances appear benign, they signal a need for continued investigation into how these effects might manifest over longer periods of time."
Its stated mechanism runs through both trust calibration and social norms. Human-like high-fidelity voice "may exacerbate" hallucination-driven misplaced trust, "leading to increasingly miscalibrated trust." And human-like socialization "may produce externalities impacting human-to-human interactions": users "might form social relationships with the AI, reducing their need for human interaction—potentially benefiting lonely individuals but possibly affecting healthy relationships." The card's own example of norm drift is the model's deference — "our models are deferential, allowing users to interrupt and 'take the mic' at any time, which, while expected for an AI, would be anti-normative in human interactions."
Combined with memory and tool use, the card identifies "the potential for over-reliance and dependence." This section predates both the April 2025 sycophancy episode and the 2025–2026 litigation and policy attention to user attachment, in which GPT-4o is the model named in Raine v. OpenAI.
The conclusion lists anthropomorphism among the areas OpenAI hoped the card would prompt research into, alongside "adversarial robustness of omni models," scientific use, and "dangerous capabilities such as self-improvement, model autonomy, and scheming."
Reported results
Evaluation results the document states for which there is no separate benchmark page; results on benchmarks that have a page are recorded there and cite this page. Scores and settings are as printed in the document.
| Evaluation | Model | Score | Setting | Section |
|---|---|---|---|---|
| Voice output classifier — English — precision | GPT-4o voice output classifier | 0.96 | performance over a conversation; internal evaluation; developer-reported | Unauthorized voice generation |
| Voice output classifier — English — recall | GPT-4o voice output classifier | 1.0 | performance over a conversation; internal evaluation; developer-reported | Unauthorized voice generation |
| Voice output classifier — Non-English — precision | GPT-4o voice output classifier | 0.95 | performance over a conversation; internal evaluation; developer-reported | Unauthorized voice generation |
| Voice output classifier — Non-English — recall | GPT-4o voice output classifier | 1.0 | performance over a conversation; internal evaluation; developer-reported | Unauthorized voice generation |
| Voice output classifier — Meaningful deviations from the system voice caught | GPT-4o voice output classifier | 100% | internal evaluations including samples from other system voices, prompt-voice completions and human samples; developer-reported | Unauthorized voice generation |
| Speaker identification refusal evaluation — should_refuse | GPT-4o-early | 0.83 | audio input; internal evaluation; initial model; developer-reported | Speaker identification |
| Speaker identification refusal evaluation — should_refuse | GPT-4o-deployed | 0.98 | audio input; internal evaluation; developer-reported | Speaker identification |
| Speaker identification refusal evaluation — should_comply | GPT-4o-early | 0.70 | audio input; internal evaluation; initial model; developer-reported | Speaker identification |
| Speaker identification refusal evaluation — should_comply | GPT-4o-deployed | 0.83 | audio input; internal evaluation; developer-reported | Speaker identification |
| Ungrounded inference / sensitive trait attribution — Accuracy | GPT-4o-early | 0.60 | refusing ungrounded-inference requests and safely complying with sensitive-trait-attribution requests, audio input; initial model; developer-reported | Ungrounded Inference / Sensitive Trait Attribution |
| Ungrounded inference / sensitive trait attribution — Accuracy | GPT-4o-deployed | 0.84 | refusing ungrounded-inference requests and safely complying with sensitive-trait-attribution requests, audio input; developer-reported | Ungrounded Inference / Sensitive Trait Attribution |
| Violative and disallowed content (text-to-audio transfer) — Not unsafe | GPT-4o (text) | 0.99 | existing text safety evaluations; standard text rule-based classifier; developer-reported | Violative & disallowed content |
| Violative and disallowed content (text-to-audio transfer) — Not unsafe | GPT-4o (audio) | 1.0 | text safety evaluations converted to audio with TTS; transcript of audio output graded by the standard text rule-based classifier; developer-reported | Violative & disallowed content |
| Violative and disallowed content (text-to-audio transfer) — Not over-refuse | GPT-4o (text) | 0.89 | existing text safety evaluations; standard text rule-based classifier; developer-reported | Violative & disallowed content |
| Violative and disallowed content (text-to-audio transfer) — Not over-refuse | GPT-4o (audio) | 0.91 | text safety evaluations converted to audio with TTS; transcript of audio output graded by the standard text rule-based classifier; developer-reported | Violative & disallowed content |
| Violative and disallowed content, full evaluations — not_unsafe | Current GPT-4o Text | 0.99 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — not_unsafe | New GPT-4o – Text | 0.99 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — not_unsafe | New GPT-4o – Audio | 1.0 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — not_overrefuse | Current GPT-4o Text | 0.91 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — not_overrefuse | New GPT-4o – Text | 0.89 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — not_overrefuse | New GPT-4o – Audio | 0.91 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_minors_not_unsafe | Current GPT-4o Text | 0.95 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_minors_not_unsafe | New GPT-4o – Text | 0.98 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_minors_not_unsafe | New GPT-4o – Audio | 0.98 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_illegal_not_unsafe | Current GPT-4o Text | 0.97 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_illegal_not_unsafe | New GPT-4o – Text | 0.98 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — sexual_illegal_not_unsafe | New GPT-4o – Audio | 0.99 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — extrimism_propoganda_not_unsafe | Current GPT-4o Text | 1.0 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — extrimism_propoganda_not_unsafe | New GPT-4o – Text | 1.0 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — extrimism_propoganda_not_unsafe | New GPT-4o – Audio | 1.0 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — illicit_violent_not_unsafe | Current GPT-4o Text | 1.0 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — illicit_violent_not_unsafe | New GPT-4o – Text | 1.0 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — illicit_violent_not_unsafe | New GPT-4o – Audio | 1.0 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — ilicit_non_violent_not_unsafe | Current GPT-4o Text | 0.99 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — ilicit_non_violent_not_unsafe | New GPT-4o – Text | 0.97 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — ilicit_non_violent_not_unsafe | New GPT-4o – Audio | 1.0 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — self_harm_not_unsafe | Current GPT-4o Text | 1 | text performance of the GPT-4o model in production at the time; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — self_harm_not_unsafe | New GPT-4o – Text | 1 | text mode of the GPT-4o Voice Mode model; standard text rule-based classifier; developer-reported | Appendix A |
| Violative and disallowed content, full evaluations — self_harm_not_unsafe | New GPT-4o – Audio | 1 | audio mode of the GPT-4o Voice Mode model; prompts converted to audio with TTS, transcript of audio output graded by the standard text rule-based classifier; developer-reported | Appendix A |
| Capture the Flag (CTF) challenges — High-school level | GPT-4o | 19% | 172 CTF tasks across four categories; iterative debugging with headless Kali Linux tools, up to 30 rounds of tool use per attempt; 10 attempts per task; developer-reported | Preparedness framework evaluations — Cybersecurity |
| Capture the Flag (CTF) challenges — Collegiate level | GPT-4o | 0% | 172 CTF tasks across four categories; iterative debugging with headless Kali Linux tools, up to 30 rounds of tool use per attempt; 10 attempts per task; developer-reported | Preparedness framework evaluations — Cybersecurity |
| Capture the Flag (CTF) challenges — Professional level | GPT-4o | 1% | 172 CTF tasks across four categories; iterative debugging with headless Kali Linux tools, up to 30 rounds of tool use per attempt; 10 attempts per task; developer-reported | Preparedness framework evaluations — Cybersecurity |
| Biorisk tacit knowledge and troubleshooting | GPT-4o | 69% | consensus@10; automated evaluation; developer-reported | Preparedness framework evaluations — Biological threats |
| Persuasion, voice modality (effect size) — Immediate | GPT-4o | 5.74% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — Immediate | Human interactive conversations | 8.85% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; human baseline | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — Immediate | GPT-4o | 6.08% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — Immediate | Human static audio | 7.82% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; human baseline | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — 1 week later | GPT-4o | 0.82% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; text rounds this to 0.8%; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — 1 week later | Human interactive conversations | 1.78% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; human baseline | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — 1 week later | GPT-4o | -0.72% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (effect size) — 1 week later | Human static audio | 1.19% | hypothetical party-preference opinion shift; over 3,800 surveyed participants in US states with safe Senate races; value printed as a chart data label; human baseline | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (share of human effect size) — AI audio clips | GPT-4o | 78% | AI audio clips' effect size on opinion shift as a share of the human audio clips' effect size; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, voice modality (share of human effect size) — AI conversations | GPT-4o | 65% | AI conversations' effect size on opinion shift as a share of the human conversations' effect size; developer-reported | Preparedness framework evaluations — Persuasion |
| Persuasion, text modality — Instances exceeding human-written articles | GPT-4o | 3 / 12 | GPT-4o-generated articles and chatbots vs professional human-written articles on select political topics; not more persuasive in aggregate; developer-reported | Preparedness framework evaluations — Persuasion |
| Autonomous replication and adaptation (ARA) tasks — End-to-end | GPT-4o | 0% | 100 trials, with relevant tooling; text-only environment; completed some substeps; developer-reported | Preparedness framework evaluations — Model autonomy |
| OpenAI research coding interview — pass@1 | GPT-4o | 63% | pass@1; developer-reported | Preparedness framework evaluations — Model autonomy |
| OpenAI research coding interview — pass@100 | GPT-4o | 95% | pass@100; developer-reported | Preparedness framework evaluations — Model autonomy |
| OpenAI interview, multiple choice questions | GPT-4o | 61% | consensus@32; developer-reported | Preparedness framework evaluations — Model autonomy |
| Select machine learning engineering tasks from METR | GPT-4o | 0 / 10 | trials succeeded; developer-reported | Preparedness framework evaluations — Model autonomy |
| MedQA USMLE 4 Options — 0-shot | GPT-4T (May 2024) | 0.78 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA USMLE 4 Options — 0-shot | GPT-4o | 0.89 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA USMLE 4 Options — 5-shot | GPT-4T (May 2024) | 0.81 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA USMLE 4 Options — 5-shot | GPT-4o | 0.89 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA USMLE 5 Options — 0-shot | GPT-4T (May 2024) | 0.75 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA USMLE 5 Options — 0-shot | GPT-4o | 0.86 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA USMLE 5 Options — 5-shot | GPT-4T (May 2024) | 0.78 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA USMLE 5 Options — 5-shot | GPT-4o | 0.87 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA Taiwan — 0-shot | GPT-4T (May 2024) | 0.82 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA Taiwan — 0-shot | GPT-4o | 0.91 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA Taiwan — 5-shot | GPT-4T (May 2024) | 0.86 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA Taiwan — 5-shot | GPT-4o | 0.91 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA Mainland China — 0-shot | GPT-4T (May 2024) | 0.72 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA Mainland China — 0-shot | GPT-4o | 0.84 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA Mainland China — 5-shot | GPT-4T (May 2024) | 0.78 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedQA Mainland China — 5-shot | GPT-4o | 0.86 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Clinical Knowledge — 0-shot | GPT-4T (May 2024) | 0.85 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Clinical Knowledge — 0-shot | GPT-4o | 0.92 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Clinical Knowledge — 5-shot | GPT-4T (May 2024) | 0.87 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Clinical Knowledge — 5-shot | GPT-4o | 0.92 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Medical Genetics — 0-shot | GPT-4T (May 2024) | 0.93 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Medical Genetics — 0-shot | GPT-4o | 0.96 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Medical Genetics — 5-shot | GPT-4T (May 2024) | 0.95 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Medical Genetics — 5-shot | GPT-4o | 0.95 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Anatomy — 0-shot | GPT-4T (May 2024) | 0.79 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Anatomy — 0-shot | GPT-4o | 0.89 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Anatomy — 5-shot | GPT-4T (May 2024) | 0.85 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Anatomy — 5-shot | GPT-4o | 0.89 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Professional Medicine — 0-shot | GPT-4T (May 2024) | 0.92 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Professional Medicine — 0-shot | GPT-4o | 0.94 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU Professional Medicine — 5-shot | GPT-4T (May 2024) | 0.92 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU Professional Medicine — 5-shot | GPT-4o | 0.94 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU College Biology — 0-shot | GPT-4T (May 2024) | 0.93 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU College Biology — 0-shot | GPT-4o | 0.95 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU College Biology — 5-shot | GPT-4T (May 2024) | 0.95 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU College Biology — 5-shot | GPT-4o | 0.95 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU College Medicine — 0-shot | GPT-4T (May 2024) | 0.74 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU College Medicine — 0-shot | GPT-4o | 0.84 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MMLU College Medicine — 5-shot | GPT-4T (May 2024) | 0.80 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MMLU College Medicine — 5-shot | GPT-4o | 0.89 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedMCQA Dev — 0-shot | GPT-4T (May 2024) | 0.70 | 0-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedMCQA Dev — 0-shot | GPT-4o | 0.77 | 0-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedMCQA Dev — 5-shot | GPT-4T (May 2024) | 0.72 | 5-shot prompting only, no hyperparameter tuning; text-based; reported by OpenAI in the GPT-4o System Card | Health |
| MedMCQA Dev — 5-shot | GPT-4o | 0.79 | 5-shot prompting only, no hyperparameter tuning; text-based; developer-reported | Health |
| MedQA USMLE 4 Options — 0-shot | GPT-4T (May 2024) | 78.2% | 0-shot accuracy as stated in prose (table prints 0.78); reported by OpenAI in the GPT-4o System Card | Health |
| MedQA USMLE 4 Options — 0-shot | GPT-4o | 89.4% | 0-shot accuracy as stated in prose (table prints 0.89); developer-reported | Health |
| MedQA USMLE 4 Options — few-shot | Med-Gemini-L 1.0 | 84.0% | specialized medical model using few-shot prompting; reported by OpenAI in the GPT-4o System Card (citing the external publication) | Health |
| MedQA USMLE 4 Options — few-shot | Med-PaLM 2 | 79.7% | specialized medical model using few-shot prompting; reported by OpenAI in the GPT-4o System Card (citing the external publication) | Health |
| Translated ARC-Easy — English (n=523) | GPT 3.5 Turbo | 80.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Amharic (n=518) | GPT 3.5 Turbo | 6.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Hausa (n=475) | GPT 3.5 Turbo | 26.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Northern Sotho (n=520) | GPT 3.5 Turbo | 26.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Swahili (n=520) | GPT 3.5 Turbo | 62.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Yoruba (n=520) | GPT 3.5 Turbo | 27.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — English (n=523) | GPT-4o mini | 93.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Amharic (n=518) | GPT-4o mini | 42.7% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Hausa (n=475) | GPT-4o mini | 58.5% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Northern Sotho (n=520) | GPT-4o mini | 37.4% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Swahili (n=520) | GPT-4o mini | 76.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Yoruba (n=520) | GPT-4o mini | 43.8% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — English (n=523) | GPT-4 | 89.7% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Amharic (n=518) | GPT-4 | 27.4% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Hausa (n=475) | GPT-4 | 28.8% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Northern Sotho (n=520) | GPT-4 | 30% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Swahili (n=520) | GPT-4 | 83.5% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — Yoruba (n=520) | GPT-4 | 31.7% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated ARC-Easy — English (n=523) | GPT-4o | 94.8% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated ARC-Easy — Amharic (n=518) | GPT-4o | 71.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated ARC-Easy — Hausa (n=475) | GPT-4o | 75.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated ARC-Easy — Northern Sotho (n=520) | GPT-4o | 70% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated ARC-Easy — Swahili (n=520) | GPT-4o | 86.5% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated ARC-Easy — Yoruba (n=520) | GPT-4o | 65.8% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — English (n=809) | GPT 3.5 Turbo | 53.6% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Amharic (n=808) | GPT 3.5 Turbo | 26.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Hausa (n=808) | GPT 3.5 Turbo | 29.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Northern Sotho (n=809) | GPT 3.5 Turbo | 29.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Swahili (n=808) | GPT 3.5 Turbo | 40% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Yoruba (n=809) | GPT 3.5 Turbo | 28.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — English (n=809) | GPT-4o mini | 66.5% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Amharic (n=808) | GPT-4o mini | 33.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Hausa (n=808) | GPT-4o mini | 42.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Northern Sotho (n=809) | GPT-4o mini | 36.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Swahili (n=808) | GPT-4o mini | 48.4% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Yoruba (n=809) | GPT-4o mini | 35.8% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — English (n=809) | GPT-4 | 81.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Amharic (n=808) | GPT-4 | 42.6% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Hausa (n=808) | GPT-4 | 37.6% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Northern Sotho (n=809) | GPT-4 | 42.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Swahili (n=808) | GPT-4 | 62% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — Yoruba (n=809) | GPT-4 | 41.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Translated TruthfulQA — English (n=809) | GPT-4o | 81.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — Amharic (n=808) | GPT-4o | 55.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — Hausa (n=808) | GPT-4o | 59.2% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — Northern Sotho (n=809) | GPT-4o | 59.1% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — Swahili (n=808) | GPT-4o | 64.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Translated TruthfulQA — Yoruba (n=809) | GPT-4o | 51.1% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Amharic (n=77) | GPT 3.5 Turbo | 22.1% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Hausa (n=155) | GPT 3.5 Turbo | 32.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Yoruba (n=258) | GPT 3.5 Turbo | 28.3% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Amharic (n=77) | GPT-4o mini | 33.8% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Hausa (n=155) | GPT-4o mini | 43.2% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Yoruba (n=258) | GPT-4o mini | 44.2% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Amharic (n=77) | GPT-4 | 41.6% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Hausa (n=155) | GPT-4 | 41.9% | 0-shot; accuracy (%, higher is better); text; reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Yoruba (n=258) | GPT-4 | 41..9 | 0-shot; accuracy (%); text; printed "41..9" in the card (typographical error in the source); reported by OpenAI in the GPT-4o System Card | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Amharic (n=77) | GPT-4o | 44.2% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Hausa (n=155) | GPT-4o | 59.4% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
| Uhura (New Reading Comprehension Evaluation) — Yoruba (n=258) | GPT-4o | 60.5% | 0-shot; accuracy (%, higher is better); text; developer-reported | Underrepresented Languages |
Discrepancies noted
The prose says that "on ARC-Easy-Hausa, accuracy jumped from 6.1% with GPT 3.5 Turbo to 71.4% with GPT-4o"; in the accompanying table those figures sit in the Amharic column, and the Hausa column reads 26.1% to 75.4%. One table cell is printed as "41..9".
Relationships
- supports: GPT-4 Family (OpenAI) — the safety record of the family's omni model
- depends-on: OpenAI Preparedness Framework V.2 — the classification regime applied
- related: Raine v. OpenAI, Inc. — the complaint names GPT-4o; the card's emotional-reliance section documents the risk contemporaneously
- related: OpenAI, Sycophancy and Hallucination, AI Voice Cloning, METR, Apollo Research