This is the OpenAI paper introducing the SimpleQA benchmark, written by Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman and William Fedus, with Wei and Karina as equal contributors, and posted to arXiv as 2411.04368 in November 2024. It presents a benchmark of 4,326 short, fact-seeking questions designed to measure whether language models "know what they know," and releases it through OpenAI's simple-evals repository.
Summary
Design goals
The authors frame factuality as hard to measure because arbitrary claims are difficult to verify and long completions contain many claims; SimpleQA sidesteps this by using only short questions with a single answer, at the acknowledged cost of leaving open whether short-form factuality generalizes to long-form. The benchmark was built for four properties: high correctness (reference answers determined by two independent AI trainers), good researcher experience (short questions, fast grading through a model API, low run-to-run variance from 4,326 items), difficulty for frontier models (GPT-4o and Claude both score below 50%, unlike the saturated TriviaQA and Natural Questions), and topical diversity.
Question criteria and collection
Trainers wrote questions to five rules. Each must have a single indisputable answer, which required specifying the scope ("which city" rather than "where"; "what year" rather than "when"). Answers must not change over time, which for television, games and sports meant pinning a point in time (for example a specific season), while "as of 2023" phrasings were disallowed. Answers must be supported by evidence: the writer supplied a supporting URL, and a second trainer answered the question independently, with only matching pairs kept. Questions must be challenging: the writer reviewed four completions from OpenAI models and could proceed only if at least one was incorrect; for most of the process all four were GPT-4 models of different release dates, and near the end one was switched to GPT-3.5 to admit slightly easier items. Finally, questions had to be answerable as of 31 December 2023.
Quality controls included few-shot ChatGPT classifiers that flagged criteria violations (missing units, time-varying answers, multiple answers) for revision, a light ChatGPT rewrite for grammar, yes/no checks by the second trainer on whether the question was timeless and indisputable, removal of questions on which the two trainers disagreed, and a requirement that the 2–4 sources across both trainers span at least two distinct website domains.
Error-rate estimate
After finalization, a third trainer answered 1,000 randomly selected questions and scored 94.4% according to the prompted grader. Of the 56 answers graded incorrect, manual review found 15 to be grader false negatives; of the remaining 41 (4.1%), seven were incomplete answers and six contradicted the source the trainer cited, leaving 2.8% that "revealed real issues with the data." The authors therefore estimate the benchmark's error rate at around 3%, assuming no grader false positives. The most common issues were ambiguous questions, reputable sources giving conflicting information, and questions with more than one correct answer.
Composition
Topics tagged post hoc by ChatGPT were science and technology (858 questions), politics (709), art (550), other (475), geography (424), sports (368), music (341), TV shows (293), history (173) and video games (135). By answer type, 32.8% were dates, 24.1% people, 15.3% numbers, 9.9% places and 18.0% other. Wikipedia was a source for about 3,500 of the 4,326 questions, followed by fandom.com (410), ac.uk (154) and imdb.com (121).
Grading and metrics
A prompted ChatGPT classifier compares the predicted and reference answers and grades each response correct (fully contains the reference without contradicting it), incorrect (contradicts it in any way, even hedged), or not attempted (does not give the reference answer and does not contradict it). The authors report only two disagreements with the grader among 300 manually read completions. Two metrics follow: overall correct (percentage of all questions answered correctly) and correct given attempted (percentage correct among attempted questions), combined into an F-score as their harmonic mean. The paper notes a loophole, credited to Adam Kalai: when a model's accuracy is below 50%, guessing whenever it is at least 50% sure always raises the F-score. It proposes as an alternative a penalty score in which a correct answer earns 1, an unattempted question 0 and an incorrect answer −p; at p = 9 a model scores positively only if at least 90% of its attempted answers are correct, "a bar that none of the models we evaluate in this paper currently meet."
Results
Table 3 reports correct / not attempted / incorrect / correct-given-attempted / F-score: Claude-3-haiku 5.1 / 75.3 / 19.6 / 20.6 / 8.2; Claude-3-sonnet 5.7 / 75.0 / 19.3 / 22.9 / 9.2; Claude-3-opus 23.5 / 39.6 / 36.9 / 38.8 / 29.3; Claude-3.5-sonnet 28.9 / 35.0 / 36.1 / 44.5 / 35.0; GPT-4o-mini 8.6 / 0.9 / 90.5 / 8.7 / 8.6; GPT-4o 38.2 / 1.0 / 60.8 / 38.0 / 38.4; o1-mini 8.1 / 28.5 / 63.4 / 11.3 / 9.4; o1-preview 42.7 / 9.2 / 48.1 / 47.0 / 44.8. The authors note that larger models outperform smaller ones, that Claude models decline to answer far more often than GPT-4o models (so that Claude 3.5 Sonnet answers fewer questions correctly than GPT-4o but reaches a similar F-score), and that Claude's scores being "not super high" suggests the GPT-4-adversarial collection produced a dataset that is hard for frontier models generally.
Calibration
Two calibration measurements are reported. Asking models to state a confidence percentage shows a positive correlation between stated confidence and accuracy, with o1-preview more calibrated than o1-mini and GPT-4o more than GPT-4o-mini, but all models fall well below the diagonal and "consistently overstate their confidence." Asking the same question 100 times at temperature 1 and treating the frequency of the most common answer as confidence shows accuracy rising with frequency, with o1-preview the best calibrated.
Stated limitation
SimpleQA "only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer"; whether this correlates with long-form factuality is left open.
Key claims
- 4,326 questions, each with a single evidence-backed answer agreed by two independent trainers, answerable as of 31 December 2023 and adversarially selected so that at least one of four OpenAI model completions was wrong.
- Estimated label error rate of about 3%, from a third-trainer audit of 1,000 questions.
- Grading is correct / incorrect / not attempted by a prompted ChatGPT classifier; the headline metric is the F-score of overall correct and correct given attempted.
- GPT-4o scored 38.2% correct (F-score 38.4), o1-preview 42.7% (44.8), Claude 3.5 Sonnet 28.9% (35.0) with 35.0% not attempted.
- Models overstate stated confidence; answer frequency across 100 samples tracks accuracy, best for o1-preview.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2411.04368 (arxiv.org); the abstract page is arxiv.org. The benchmark is at github.com.
Relationships
- supports: SimpleQA — the primary text for the benchmark's construction, grading scheme and baseline results.
- supports: Humanity's Last Exam — HLE adopts this paper's stated-confidence calibration setup.
- related: SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., 2025) — Google's filtered 1,000-question derivative, which critiques the label noise, redundancy and topical skew described here.
- related: OpenAI
- instance-of: AI Benchmarks and Evaluation