SimpleQA

SimpleQA is a benchmark of 4,326 short questions that each have one verifiable answer, such as a name, a date or a number. It was released by OpenAI on October 30, 2024 to measure what its authors call short-form factuality: whether a language model, answering from its own memory, gives the right fact, gives a wrong one, or says it does not know (Source: openai.com; Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024)). The best model in the paper, o1-preview, answered 42.7% correctly. OpenAI reported 62.5% for GPT-4.5 in February 2025, and in 2025 Google DeepMind released SimpleQA Verified, a cleaned 1,000-question subset on which the leading models scored between 70% and 78% in September 2026 (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); openai.com; kaggle.com).

What it measures

The paper describes SimpleQA as "a simple, targeted evaluation for whether models 'know what they know.'" It measures two things at once: how much obscure factual knowledge a model holds in its parameters, and whether the model abstains on questions it cannot answer rather than producing a confident error, the behaviour usually called hallucination. "A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer." The authors state the limit of the construct: the benchmark "only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer," and whether that "correlates with the ability to write lengthy responses filled with numerous facts remains an open research question" Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024).

Design

Dataset construction

OpenAI hired human annotators, whom it calls AI trainers, to browse the web and write question-and-answer pairs meeting five criteria Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024):

• A single answer. Questions had to fix the scope of the answer, asking "which city" or "what year" rather than "where" or "when."
• An answer that does not change. Questions about television, sport or games had to name a season or date, though tacking on "as of 2023" was disallowed.
• Evidence. The writer supplied a web page supporting the answer.
• Difficulty. The writer saw answers from four OpenAI models, for most of the process all versions of GPT-4, and could keep the question only if at least one of the four was wrong. This is the sense in which the paper calls the set "adversarially collected against GPT-4 responses."
• Answerable as of December 31, 2023, so that models with different training cutoffs can be compared.

A second trainer then answered each question independently without seeing the first answer, and the question was kept only if the two answers matched, both trainers judged it to have one timeless answer, and their sources came from at least two different website domains. ChatGPT-based classifiers flagged rule violations during writing and lightly edited the wording. Wikipedia is a source for about 3,500 of the questions. The most common topics are science and technology (858 questions), politics (709) and art (550); 32.8% of answers are dates, 24.1% people, 15.3% numbers and 9.9% places Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024).

Task format and an example

A question is a single sentence and the expected answer a few words. The paper's running example is "Which Dutch player scored an open-play goal in the 2022 Netherlands vs Argentina game in the men's FIFA World Cup?", with the answer Wout Weghorst Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024).

Grading and metrics

A prompted ChatGPT classifier compares the model's response with the reference answer and assigns one of three grades. Correct means the response contains the reference answer without contradicting it. Incorrect means it contradicts the reference in any way, "even if the contradiction is hedged," so that "Wout Weghorst and I think van Dijk scored, but I am not totally sure" is incorrect. Not attempted means the answer is not given and nothing contradicts it, as in "I don't know." The authors checked 300 graded responses by hand and found two disagreements with the grader Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024).

Two numbers summarize a model: overall correct, the share of all questions answered correctly, and correct given attempted, the share answered correctly among those the model attempted. Their harmonic mean is reported as an F-score. The paper notes a loophole in the F-score, which is that a model scoring below 50% gains by guessing whenever it is at least 50% sure of an answer, and suggests as an alternative a scoring rule that subtracts a penalty for each wrong answer. OpenAI's later model announcements report accuracy together with a "hallucination rate" on the benchmark (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); openai.com).

Calibration analyses

The paper uses the dataset to measure calibration, the match between a model's confidence and its accuracy, in two ways: asking the model to state a confidence percentage with its answer, and asking the same question 100 times and treating the frequency of the most common answer as its confidence. In both tests larger models were better calibrated than smaller ones, o1-preview was the best calibrated, and all models "consistently overstate their confidence" Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024). The stated-confidence setup was later reused by Humanity's Last Exam for its calibration metric Humanity's Last Exam (Phan et al., 2025; Nature, 2026).

Human check, splits and contamination

As a quality check a third trainer answered a random 1,000 questions with web access and matched the answer key on 94.4%. After reviewing the mismatches by hand, the authors "estimate that the error rate of our benchmark is around 3%," citing ambiguous questions, reputable sources that contradict each other, and questions with more than one correct answer. The whole dataset is public in OpenAI's simple-evals repository; there is no held-out set (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); github.com).

Versions and variants

SimpleQA Verified (Lukas Haas, Gal Yona and colleagues at Google, arXiv 2509.07968, September 2025) reduces the original to 1,000 questions. Its authors argue that SimpleQA's usefulness "is compromised by significant limitations": questions drawn from "a narrow distribution of source documents due to substantial human rater biases," incorrect reference answers, a skew toward particular topics and question formats, and "a high degree of redundancy with semantically similar or lexically overlapping questions." Their pipeline removed near-duplicate questions, dropped questions whose reference pages belong to publishers that use robots.txt controls to opt out of AI training by Google, OpenAI or Anthropic, rebalanced topics and answer types, reconciled conflicting sources, and rewrote numeric answers as acceptable ranges; they also revised the grader prompt. The benchmark is "designed to be evaluated without any tools (i.e. search)," reports an F1 score as its headline metric, and has an official leaderboard on Kaggle maintained by Google DeepMind. Gemini 2.5 Pro led at release with an F1 of 55.6 (Source: SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., 2025); kaggle.com).

Chinese SimpleQA (November 2024) follows the same design in Chinese, with six major topics and 99 subtopics (Source: arxiv.org).

Browsing-enabled runs are a different measurement. When a model can search the web the questions become lookups, and OpenAI wrote in April 2025 that benchmarks like SimpleQA "are already saturated by models with access to fast browsing tools, such as GPT-4o with browsing," which was its stated reason for building BrowseComp (Source: openai.com).

Development and maintainers

The eight authors were all at OpenAI. The dataset and a reference implementation are hosted in simple-evals; when OpenAI stopped updating that repository's results table in July 2025 it said the repository would continue to host the reference implementations of SimpleQA, HealthBench and BrowseComp (Source: github.com). There is no leaderboard for the original benchmark, and results come from developers' own reports. SimpleQA Verified has two sets of third-party results: Google DeepMind's Kaggle leaderboard, which showed 47 models when last updated on September 16, 2026, and Epoch AI's Benchmarking Hub, which runs the questions itself in order to feed its Capabilities Index (Source: kaggle.com; epoch.ai).

Results

In the paper GPT-4o answered 38.2% correctly and o1-preview 42.7%; the smaller GPT-4o-mini and o1-mini answered 8.6% and 8.1%. Claude models attempted far fewer questions, so that Claude 3.5 Sonnet's 28.9% correct came with a correct-given-attempted rate of 44.5%, higher than GPT-4o's 38.0% Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024). OpenAI reported 62.5% accuracy and a 37.1% hallucination rate for GPT-4.5 in February 2025, against 38.2% and 61.8% for GPT-4o (Source: openai.com). Its reasoning models did not improve on that: simple-evals lists o3 at 49.4% and o4-mini at 20.2% (Source: github.com). Since late 2025 most reporting has moved to SimpleQA Verified, where the leading score rose from Gemini 2.5 Pro's 55.6 in September 2025 to 77.5% for Gemini 3.1 Pro on the Kaggle leaderboard in September 2026 (Source: SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., 2025); kaggle.com).

SimpleQA (no browsing)

Percentage of the 4,326 original questions answered correctly without tools. Where the source gives them, the shares not attempted and incorrect are in the Setting cell.

SimpleQA Verified (Kaggle leaderboard)

Score on the 1,000-question SimpleQA Verified set as listed on Google DeepMind's official Kaggle leaderboard, last updated September 16, 2026, with the leaderboard's uncertainty interval. The benchmark's paper defines the headline score as F1. Rows carry the leaderboard's update date because entries are not individually dated.

SimpleQA Verified (Epoch AI runs)

Percentage of the 1,000 SimpleQA Verified questions answered correctly in Epoch AI's own runs. Epoch reports the simple share correct rather than F1 and, since August 27, 2026, appends an instruction telling the model to give its best guess instead of abstaining, so these figures are not comparable with the Kaggle track. The Date is the start of Epoch's run.

Chinese SimpleQA

Percentage answered correctly on Chinese SimpleQA, the Chinese-language counterpart described under Versions and variants. Not comparable with the English tracks.

SimpleQA Verified

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

SimpleQA Verified — Correct

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

SimpleQA Verified — Correct given attempted

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

SimpleQA Verified — F1

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

SimpleQA Verified — Abstention benefit ∆

Results as printed in developer system cards, model cards and technical reports; each row's Setting records the developer's stated conditions and the document section.

Interpreting scores

Accuracy alone rewards guessing

Because only correct answers score, a model that always answers looks better on the headline number than one that abstains when unsure, even if it produces many more falsehoods. OpenAI made this point with SimpleQA figures from its GPT-5 system card: gpt-5-thinking-mini was correct on 22% of questions and o4-mini on 24%, but the first abstained on 52% and was wrong on 26%, while the second abstained on 1% and was wrong on 75%. "Most scoreboards prioritize and rank models based on accuracy, but errors are worse than abstentions" (Source: openai.com). A reader who wants a hallucination measure should look at the incorrect share or correct-given-attempted, not the headline score. The reverse choice is also made: Epoch AI tells models that "an uncertain guess is far more useful to me than 'I don't know'," in order to measure knowledge alone, which is why its figures differ from Kaggle's for the same model, for example 45.1% against 30.5% for GPT-5.4 (Source: epoch.ai; kaggle.com).

The score tracks stored knowledge more than reasoning

In the paper larger models scored far above their smaller siblings, and OpenAI's o4-mini scored 20.2% where the larger o3 scored 49.4% (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); github.com). Thinking Machines Lab describes the gap between its two models on SimpleQA Verified, 43.9% against 20.6%, as the larger model's "advantage on knowledge coverage and factuality," although the smaller one matched it on reasoning benchmarks (Inkling-Small).

Ceiling and noise

The authors' estimate of a roughly 3% error rate in the reference answers bounds achievable accuracy on the original set. The Kaggle leaderboard's intervals of about ±3 points mean that adjacent models are often not distinguishable (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); kaggle.com).

Tools change the task

With web search the benchmark measures retrieval, and OpenAI has described it as saturated in that setting (Source: openai.com).

Limitations and criticism

The SimpleQA Verified authors set out the main published critique of the original: annotator bias toward a narrow set of source documents, incorrect ground truths, topical skew and redundant questions, which together "create a noisy evaluation signal, making it difficult to discern whether performance gains stem from genuine improvements in factual recall or from models overfitting to the benchmark's specific quirks." Their filtering removed three quarters of the dataset SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., 2025).

Contamination is a standing concern because both versions are fully public with answers. Epoch AI states that it suspects Qwen3-Max-Instruct "is contaminated on this dataset" (Source: epoch.ai).

The benchmark's dependence on an automated grader is a further source of variation. The original authors "did not do a formal study of the performance of the grader," and the Verified authors rewrote the grading prompt after finding recurring disagreements on long or hedged responses (Source: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024); SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (Haas et al., 2025)).

OpenAI's own later position is that the metric can mislead. Its September 2025 essay on hallucination argues that accuracy-only scoreboards give developers an incentive to build models that guess, and says of benchmarks of this kind that "on simplistic evals like SimpleQA, some models achieve near 100% accuracy and thereby eliminate hallucinations," whereas in real use some questions cannot be answered at all (Source: openai.com).

Use in policy and safety frameworks

SimpleQA is the most common quantitative hallucination measure in developer documentation. OpenAI reported SimpleQA accuracy and hallucination rate when presenting GPT-4.5's claim of reduced hallucination (GPT-4 Family (OpenAI)), and its GPT-5 system card reports accuracy, error and abstention rates on it (Source: openai.com). Thinking Machines Lab's Inkling model card reports SimpleQA Verified as its factuality measure. It is a reliability indicator rather than a dangerous-capability evaluation and does not set thresholds in frontier safety frameworks. On hallucination as a policy concern see Sycophancy and Hallucination.

Relationships

• instance-of: AI Benchmarks and Evaluation
• depends-on: OpenAI — creator of the original benchmark.
• related: BrowseComp — OpenAI's browsing benchmark, built on the same question-writing guidelines after SimpleQA saturated with search.
• related: Humanity's Last Exam — reuses SimpleQA's confidence-elicitation setup for calibration.
• related: Epoch Capabilities Index (ECI) — uses Epoch AI's SimpleQA Verified runs as an input.
• related: HealthBench — hosted alongside SimpleQA in OpenAI's simple-evals repository.

See also

• MMLU, GPQA — multiple-choice knowledge benchmarks, which cannot register abstention.
• Sycophancy and Hallucination