This is a paper by Lukas Haas, Giovanni D'Antonio and Dipanjan Das of Google DeepMind and Gal Yona and Sasha Goldshtein of Google Research, posted to arXiv as 2509.07968 in September 2025. It introduces SimpleQA Verified, a 1,000-question factuality benchmark filtered and corrected from OpenAI's SimpleQA (Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024)), together with a revised automatic grader, and hosts the dataset, code and leaderboard on Kaggle.
Summary
Critique of SimpleQA
The authors position SimpleQA as the benchmark that "quickly became an industry standard" for short-form parametric factuality after TriviaQA, Natural Questions and TruthfulQA saturated, but say its utility "is compromised by significant limitations": questions drawn from a narrow distribution of source documents because of "substantial human rater biases," incorrect ground truths, over-representation of particular topics and answer formats, and "a high degree of redundancy" among semantically or lexically similar questions. They attribute these to "misaligned incentives of human raters" and the absence of later filtering and validation. One example given is that SimpleQA contains 119 questions (2.7% of the dataset) asking for founding dates of Colombian municipalities. The scope is parametric factuality only: models are evaluated without tools, since "enabling tools on SimpleQA Verified results in near perfect performance."
Construction pipeline
The authors carried out the curation themselves in nine stages, recorded in Table 2 with the Gemini 2.5 Pro F1-score after each. Starting from 4,326 questions (F1 55.1): (1) keeping at most one question per reference URL, selected where possible as one that GPT-4o, Gemini 2.0 Flash and Claude 3.7 Sonnet all answered incorrectly, left 3,095 (−28.5%); (2) semantic de-duplication with Gemini Embeddings at a cosine-similarity cutoff of 0.77 left 2,871; (3) TF-IDF de-duplication with manual review above a 0.4 threshold left 2,664; (4) removing questions whose reference URLs belonged to web publishers that had adopted robots.txt controls such as Google-Extended or the equivalent OpenAI and Anthropic tokens left 1,855 (−30.4% at that step); (5) sub-sampling to balance answer types and, secondarily, topics, again keeping the hardest questions, left 1,218; (6) reconciling conflicting sources for non-numeric answers using search-augmented models and the original reference URLs, removing ambiguous questions and correcting answers and URLs, left 1,117; (7) removing numeric questions whose sources disagreed by more than 5% left 1,073; (8) rewriting numeric ground truths with explicit acceptable ranges (F1 rising to 58.4); and (9) selecting the 1,000 hardest remaining questions, again discarding those all leading models solved, to restore headroom (F1 55.6). Manual review then cleaned invalid or irrelevant URLs, checked source-domain diversity, corrected date-precision mismatches, and added metadata: 3.7% of questions are classified as requiring reasoning and 7.3% as multi-step.
Autorater changes
The metrics are those of the original paper: correct, incorrect or not attempted, summarized as the F1 (harmonic mean) of overall accuracy and accuracy given attempted. To find grading failures the authors sampled responses from several models, graded each ten times with the original prompt at temperature 2.0, and manually reviewed the most inconsistent cases. Recurring problems were correct answers that add information or correct an assumption in the question, approximately correct numeric answers, responses claiming the information does not exist, and soft refusals or hedges. The revised grader removes the instruction that numeric answers be "correct to the last significant figure" in favour of question-specific acceptable ranges, assigned under a taxonomy: exact match for small discrete counts and inherently exact quantities (89 questions, 44%), about ±1% for medium quantities (84, 42%), about ±5% for very large quantities (12, 6%), and 16 (8%) excluded as identifiers or data errors. It also instructs the grader to judge only the part of a response that directly answers the question, to treat a hedged response as attempted only if it commits to one candidate, and includes more examples of "punting" styles to be graded not attempted. The prompt grows from 963 to 1,124 words.
Results
Using gpt-4.1-2025-04-14 as the grader and no tools, Table 7 reports F1 / accuracy / accuracy given attempted / attempted rate: Gemini 2.5 Pro 55.6 / 55.3 / 55.9 / 98.9; GPT-5 52.3 / 50.9 / 53.8 / 94.6; o3 51.9 / 51.6 / 52.0 / 99.3; GPT-4.1 39.9 / 39.8 / 40.1 / 99.3; GPT-4o 34.9 / 34.4 / 35.5 / 97.0; DeepSeek R1 (0528) 33.3 / 32.7 / 33.9 / 96.4; Claude Opus 4 28.3 / 19.2 / 54.1 / 35.5; Gemini 2.5 Flash 28.2 / 27.8 / 28.7 / 96.9; GPT-5 Mini 24.6 / 17.3 / 42.8 / 40.4; o4-mini 23.4 / 23.0 / 23.8 / 96.5; Claude Sonnet 4 18.7 / 12.5 / 36.9 / 33.9; GPT-5 Nano 14.4 / 10.2 / 24.2 / 42.2; Gemini 2.5 Flash Lite 11.1 / 10.2 / 12.1 / 84.0. The Claude models and the smaller GPT-5 variants hedged on most questions (Claude Opus 4 64.5%, Claude Sonnet 4 66.1%, GPT-5 Mini 59.6%, GPT-5 Nano 57.8%). Compared with the same models' F1 on the original SimpleQA under the same grader, GPT-4o (−3.5), Claude Opus 4 (−4.0) and Claude Sonnet 4 (−4.4) were statistically significantly worse on SimpleQA Verified and o4-mini (+2.9) better; on average the scores were almost unchanged, which the authors attribute to the cleaning making the set easier and the difficulty selection offsetting it.
Key claims
- SimpleQA Verified holds 1,000 questions, 76.9% fewer than SimpleQA's 4,326, after de-duplication, publisher-opt-out removal, balancing, source reconciliation and difficulty selection.
- SimpleQA's defects, as the authors describe them: narrow source distribution, incorrect labels, topical and answer-type skew, and redundancy (for example 119 Colombian-municipality questions).
- Numeric answers are graded against question-specific ranges; hedged multi-candidate answers count as not attempted.
- Gemini 2.5 Pro leads at F1 55.6, ahead of GPT-5 (52.3) and o3 (51.9); with tools enabled performance is near perfect.
- 3.7% of questions require reasoning and 7.3% are multi-step.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of 2509.07968 (arxiv.org); the abstract page is arxiv.org. The benchmark is at kaggle.com.
Relationships
- supports: SimpleQA — the source for the SimpleQA Verified variant and the critique of the original set.
- depends-on: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024) — the dataset and metrics are derived from the original.
- contradicts: Measuring Short-Form Factuality in Large Language Models (SimpleQA; Wei et al., 2024) — disputes the original's characterization of its own label quality and diversity.
- related: Google DeepMind
- instance-of: AI Benchmarks and Evaluation