This is the paper introducing the BrowseComp benchmark, by Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese of OpenAI; Wei, Sun and Papay are marked as equal contributors. It was posted to arXiv as 2504.12516 on 16 April 2025. The dataset is distributed through OpenAI's simple-evals repository.
Summary
BrowseComp, short for "Browsing Competition," comprises 1,266 questions "that require persistently navigating the internet in search of hard-to-find, entangled information." The authors present it as analogous to programming competitions for coding agents: an incomplete but useful proxy that "sidesteps challenges of a true user query distribution, like generating long answers or resolving ambiguity," while measuring "the important core capability of exercising persistence and creativity in finding information." They argue earlier retrieval benchmarks target information easily found and have become saturated.
Construction
Questions were written entirely by human trainers under instructions that largely followed SimpleQA: fact-seeking questions with single, indisputable, short answers that would not change over time and were supported by evidence. Trainers were encouraged to construct questions in inverted fashion — start from a "seed" such as a person, event or artifact, collect several characteristics with a large search space, and pose a question that is hard to find but easy to verify. The exemplar asks for the title of an EMNLP paper from 2018–2023 whose first author studied at Dartmouth and fourth author at the University of Pennsylvania.
Three checks enforced difficulty: trainers verified that GPT-4o (with and without browsing), OpenAI o1 and an early version of Deep Research could not solve the question; ran five simple Google searches to confirm the answer was not on the first results pages; and aimed for questions another person could not solve within ten minutes. The last check was not strictly enforced, but for a portion of questions a second trainer attempted them, and trainers whose questions were solved more than 40% of the time were asked to revise. The authors acknowledge that inverted construction means reference answers are "likely but not guaranteed" to be unique; trainers were asked to add criteria until confident, and a second trainer finding a different valid answer within ten minutes triggered revision. Topics were chosen by trainers' personal interests; a post-hoc classification gives TV shows and movies 16.2% (205), other 15.6%, science and technology 13.7%, art 10.0%, history 9.9%, sports 9.7%, music 9.2%, video games 5.6%, geography 5.5% and politics 4.7%. The dataset originally had 1,287 questions; after reviewing the 118 on which Deep Research had a 0% pass rate over 64 trials, 21 with mismatched, ambiguous or incorrect reference answers were removed. The paper embeds a canary string and asks that examples not be posted online.
Grading
Because every reference answer is a short string, an AI model judges whether a predicted answer is semantically equivalent to the reference, using the same grading prompt as Humanity's Last Exam. Models are also asked for a confidence score, from which calibration error is computed.
Human performance
Trainers from the same pool, excluding each question's author and without AI assistants, attempted 1,255 of the 1,266 questions. They gave up on 888 (70.8%) after about two hours and solved 367 (29.2%); of the solved questions, the trainer's answer agreed with the reference on 317 (86.4%). Solved questions ranged from under an hour to two or three hours. The authors note the trainers "were also not competition-level internet browsers," and that experienced professionals such as investigative journalists with ample time might solve many of the abandoned questions.
Model results
At release, GPT-4o (gpt-4o-2024-08-06) scored 0.6% with 69% calibration error, GPT-4o with browsing 1.9% (82%), GPT-4.5 0.9% (68%), OpenAI o1 at medium reasoning effort and without browsing 9.9% (65%), and OpenAI Deep Research 51.5% (91%). A footnote states that Deep Research "is trained on data that specifically teach the model to be good at BrowsingComp tasks." The authors read o1's result as showing that some answers can be surfaced by reasoning over internal knowledge, and browsing-enabled models' higher calibration error as suggesting "access to web tools may increase the model's confidence in incorrect answers."
Accuracy for an early Deep Research version rose smoothly with browsing effort (test-time compute). Sampling 64 outputs per question and aggregating by majority vote, confidence-weighted vote or best-of-N improved performance by 15% to 25% over a single attempt, with best-of-N highest, which the authors take as evidence the model "frequently 'knows' when it's right" even though its stated confidence is poorly calibrated. Over 64 trials, Deep Research solved 16% of tasks on every attempt and failed 14% on every attempt; when given the ground-truth answer for the never-solved subset it usually retrieved supporting evidence, confirming those questions are solvable.
Key claims
- 1,266 questions (from an initial 1,287), human-written, short verifiable answers, inverted construction; canary string included.
- Human trainers: 29.2% solved within about two hours (367 of 1,255), 70.8% abandoned, 86.4% agreement with the reference on solved items.
- Release results: GPT-4o 0.6%, GPT-4o with browsing 1.9%, GPT-4.5 0.9%, o1 9.9%, Deep Research 51.5% (trained on BrowseComp-style data); calibration error 65%–91%.
- Performance scales with test-time compute; 64-sample aggregation adds 15–25 points.
- Stated limitations: not a user-query distribution; uniqueness of answers likely but not guaranteed; evidence limited to text rather than images, video, audio or interactive pages.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); the abstract page (arxiv.org) records a single version submitted 16 April 2025. The benchmark code is at github.com.
Relationships
- supports: BrowseComp — the primary text for the benchmark's construction, human baseline and release results.
- depends-on: SimpleQA — question-writing instructions followed SimpleQA's; Humanity's Last Exam — grading prompt reused.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, OpenAI