This is the paper introducing PaperBench from OpenAI. Giulio Starace (corresponding), Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias and Tejal Patwardhan are marked as equal contributors, with Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke and Amelia Glaese. It was posted to arXiv as 2504.01848 on 2 April 2025 and revised on 7 April 2025 (v3, summarised here), and is formatted as an ICML preprint. Code is released through OpenAI's preparedness repository.
Summary
PaperBench evaluates "the ability of AI agents to replicate state-of-the-art AI research." An agent receives a paper and an addendum of author clarifications and must produce a repository with a reproduce.sh entry point that, run in a fresh environment, reproduces the paper's empirical results. Agents may not use or view the authors' original code or other online replications, which are listed in per-paper blacklists; a post-hoc monitor found 10 such cases across 646 runs, which were scored zero. The rubric is hidden from the agent.
Grading
Each paper has a rubric: a tree that decomposes "the core contributions of the paper have been reproduced" into ever finer requirements until each leaf could be checked by an expert in under 15 minutes. Leaves are one of three types — Result Match (evidence in the reproduction outputs that a result was replicated), Execution (a step ran during reproduce.sh) and Code Development (the source contains a correct implementation) — so that partial progress earns credit. Nodes are weighted by importance relative to siblings; a parent's score is the weighted average of its children, and the root score is the Replication Score. Submissions are copied to a fresh Ubuntu 24.04 VM with an A10 GPU and executed (capped at 12 hours; agent scripts averaged 5.5 minutes) before grading, so hard-coded results can be distinguished from reproduced ones. Across the 20 papers there are 8,316 leaf nodes; individual rubrics range from 94 to 2,551 nodes. Rubrics were written with an original author of each paper over multiple weeks per paper, which the authors call "the most time-intensive aspect of developing PaperBench." A cheaper variant, PaperBench Code-Dev, grades only Code Development leaves without a reproduction step, cutting grading cost by about 85%; its scores correlate only weakly with the full benchmark (Pearson r = 0.48 for o1).
Because expert grading took tens of hours per paper, grading is automated by SimpleJudge, which is shown the paper, the rubric, one leaf and the ten most relevant files and returns a binary judgment. JudgeEval, built from human-graded partial replications of four PaperBench papers and one development paper, measures judges as binary classifiers: o3-mini (high) with SimpleJudge reached F1 0.83 at about $66 per paper, o1 0.84 at $830, GPT-4o 0.73 at $120, o1-mini 0.78 at $72, GPT-4o-mini 0.59 at $8, random 0.49.
Dataset
The 20 papers are ICML 2024 Spotlight and Oral papers curated for suitability, spanning LLMs, probabilistic methods, deep reinforcement learning, robustness, data-centric AI, generative models and distribution shift; two NeurIPS 2024 workshop papers form a development set and a held-out set is kept internally. Each paper has an addendum of author clarifications and, where needed, a judge-only addendum.
Results
Agents ran in Ubuntu 24.04 containers with one A10 GPU, internet access, a Hugging Face key and an OpenAI API key loaded with $1,000, using BasicAgent — a tool-use loop based on Inspect AI's basic agent with bash, Python, browser and paginated file-reader tools — for up to 12 hours, three runs per paper. Replication Scores were Claude 3.5 Sonnet (New) 21.0% ± 0.8, o1 (high) 13.2% ± 0.3, DeepSeek-R1 6.0% ± 0.3, GPT-4o 4.1% ± 0.1, Gemini 2.0 Flash 3.2% ± 0.2 and o3-mini (high) 2.6% ± 0.2; Claude 3.7 Sonnet could not be run within Anthropic rate limits. All models except Claude 3.5 Sonnet frequently stopped early, claiming completion or an insoluble problem, and none strategised about the time available. IterativeAgent, which removes the ability to end early and prompts piecemeal work, raised o1 to 24.4% ± 0.7 (26.0% ± 0.3 with a 36-hour limit) and o3-mini to 8.5% but lowered Claude 3.5 Sonnet to 16.1%, which the authors read as prompt tuning suited to o-series models. On PaperBench Code-Dev, o1 with IterativeAgent scored 43.4% ± 0.8. The authors describe these as initial baselines, not upper limits.
Human baseline
Eight ML PhD holders or students (Berkeley, Cambridge, CMU, Columbia, Cornell, Purdue, TU Wien, UMass Amherst), screened by CV and a technical test, made three independent attempts each on four papers, working part-time over four weeks with an A10 (an A100 in four cases), free to use AI assistants but not blacklisted sites, with active hours tracked; only the best performer per paper continued past the first week. Against an o1 IterativeAgent run graded at 1, 3, 6, 12 and 36 hours, o1 led early but plateaued after about an hour while humans overtook it after 24 hours, a pattern the authors relate to METR's RE-Bench findings.
Limitations stated by the authors
Only 20 papers, though thousands of requirements; the authors' code for almost all papers is online, so later models may have internalised solutions even if early-2025 models likely had not; rubric creation takes an expert several full days per paper and proved hard to train others to do; the judge is less accurate than an expert and non-deterministic; and cost is about $400 in API credits per o1 IterativeAgent 12-hour rollout, about $8,000 per full run, plus about $66 per paper to grade.
Key claims
- 20 ICML 2024 Spotlight/Oral papers; 8,316 leaf requirements; rubrics co-developed with authors; three leaf types; weighted-average Replication Score.
- SimpleJudge with o3-mini: F1 0.83 on JudgeEval at about $66 per paper.
- BasicAgent, 12 hours: Claude 3.5 Sonnet (New) 21.0%, o1 13.2%, others under 10%; IterativeAgent: o1 24.4% (26.0% at 36 hours), o3-mini 8.5%, Claude 3.5 Sonnet 16.1%; Code-Dev o1 43.4%.
- ML PhDs overtook o1 after 24 hours of tracked work on a paper subset.
- 10 blacklist violations in 646 runs, scored zero.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); the abstract page (arxiv.org) records submission on 2 April 2025 and revision on 7 April 2025, 30 pages and 14 figures.
Relationships
- supports: PaperBench — the primary text for the benchmark's task, rubrics, judge and launch results.
- related: MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (Chan et al., 2024) — the same OpenAI team's ML-engineering benchmark; RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (Wijk et al., 2024) — the human-versus-agent time pattern the authors cite.
- supports: OpenAI
- instance-of: AI Benchmarks and Evaluation