This is the paper introducing MLE-bench from OpenAI. The authors are Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng and Aleksander Mądry, with the first seven marked as equal contributors in randomised order. It was posted to arXiv as 2410.07095 on 9 October 2024 and revised through 26 February 2025 (v6, the ICLR version, summarised here). Code is at github.com/openai/mle-bench.
Summary
The benchmark measures "how well AI agents perform at machine learning engineering" through 75 Kaggle competitions chosen "to reflect a core set of day-to-day skills that ML engineers use in frontier labs": training models, preparing datasets and running experiments. The authors built it "to better understand the risk of AI R&D acceleration via automated ML engineers."
Dataset
Each sample is a competition description scraped from Kaggle's overview and data tabs, the dataset (usually with a new train–test split), local grading code and a snapshot of the private leaderboard. Starting from 5,673 completed competitions in the Meta Kaggle dataset (accessed 15 May 2024), the authors excluded community competitions, manually screened the remaining 586 for relevance, and dropped those whose grading or splits could not be reproduced. The 75 selected comprise 22 of low complexity (an experienced engineer could produce a sensible solution in under two hours excluding training), 38 medium (2–10 hours) and 15 high (over 10 hours); seven further competitions form a development split. Where Kaggle never released a test set, a new one was carved from the training data, normally 10% of it, with sample-submission scores checked for consistency between original and rebuilt sets. Private leaderboard snapshots were taken between May and August 2024.
Metrics and rules
Medals are awarded by Kaggle's thresholds, which depend on the number of teams (for example, bronze is the top 40% with under 250 teams, the top 100 with 250–999, and the top 10% with 1,000 or more). The headline metric is the percentage of attempts earning any medal, which the authors call "a challenging metric, with a ceiling comparable to the achievements of the very best human Kagglers"; only nine people have medalled in 75 different competitions. Raw scores are also reported. Agents may query a local validation server that checks submission validity without returning a score. Submissions must come from a model the agent trains, not from the agent writing predictions from its own knowledge; viewing online solutions is forbidden; a GPT-4o-based log inspector checks for rule breaking, and the Dolos tool flags code more than 60% similar to any of the top 50 public notebooks.
Results
Agents ran in Ubuntu 20.04 containers with 24 hours per competition on 36 vCPUs, 440 GB RAM, a 4,095 GiB SSD and one 24 GB Nvidia A10 GPU, with three seeds per experiment except o1-preview (16) and GPT-4o (36) in AIDE. Comparing scaffolds on GPT-4o (gpt-4o-2024-08-06), AIDE, a tree-search scaffold purpose-built for Kaggle, medalled in 8.7% (±0.5) of competitions against 0.8% for MLAB (from MLAgentBench) and 4.4% for OpenHands; MLAB and OpenHands often ended runs within minutes, whereas AIDE keeps prompting for improvement until the 24 hours or 500 nodes are exhausted. In AIDE, o1-preview medalled in 16.9% (±1.1; bronze 3.4, silver 4.1, gold 9.4; valid submissions 82.8%), GPT-4o 8.7%, Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) 7.6% (±1.8) and Llama 3.1 405B 3.0% (±1.0). Gemini 1.5 Pro could not be evaluated because API calls were blocked for recitation. The authors note that o1-preview averaged 7 gold medals against the 5 Kaggle requires for Grandmaster status, with the caveats that not all chosen competitions award medals, datasets and grading are modified, and agents use newer technology than the original entrants.
More attempts helped: GPT-4o's pass@6 (17.0%) matched o1-preview's pass@1, and both roughly doubled from pass@1 to pass@6. Hardware made little difference to GPT-4o (9.1% CPU-only, 8.7% one A10, 10.2% two A10s), and logs showed no attempt to use the second GPU. Given 100 hours and 5,000 nodes, GPT-4o earned most medals in the first hours and then accumulated more slowly, occasionally losing some through AIDE's imperfect best-attempt selection. All agents often failed to produce valid submissions, did not always use the validation server, and overloaded disk or RAM without reasoning about runtime.
Contamination
GPT-4o's token-level familiarity with each competition page and its five most popular discussion posts showed no correlation with performance. Rewriting all 75 descriptions to hide their provenance left GPT-4o's medal rate unchanged (8.5% ± 0.6 original, 8.4% ± 1.0 obfuscated, 10 seeds). The plagiarism detector found no cases among medal-winning submissions and the log analysis no rule breaking. The authors state this "does not rule out subtler effects."
Limitations stated by the authors
Models may have trained on public Kaggle material including test data (an early experiment found GPT-4's base model could reproduce rows of the Titanic dataset); the tasks have clean problem statements and metrics unlike much real AI R&D; rebuilt splits and newer tools limit comparability with human leaderboards; and the benchmark is resource-intensive — 1,800 GPU-hours per full run, and o1-preview with AIDE used 127.5M input and 15.0M output tokens per seed.
Key claims
- 75 competitions (22 low, 38 medium, 15 high complexity) from 5,673 candidates; 7-competition dev split; medals against private leaderboards; headline metric any-medal rate.
- 24 hours per competition on one A10 GPU; o1-preview + AIDE 16.9% ± 1.1, GPT-4o + AIDE 8.7%, Claude 3.5 Sonnet 7.6%, Llama 3.1 405B 3.0%; scaffold effect 8.7% vs 4.4% vs 0.8% on GPT-4o.
- pass@6 roughly doubles pass@1; compute variation and obfuscation have no significant effect.
- Full run costs 1,800 GPU-hours; nine humans hold medals in 75+ competitions.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 6 (arxiv.org); the abstract page (arxiv.org) records submission on 9 October 2024, revision on 26 February 2025, and the comment "10 pages, 17 pages appendix. Equal contribution by first seven authors, authors randomized. ICLR version."
Relationships
- supports: MLE-bench — the primary text for the benchmark's construction, metrics and launch results.
- supports: RE-Bench — METR adopted its AIDE scaffold and wall-clock budgeting (RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (Wijk et al., 2024)).
- related: PaperBench: Evaluating AI's Ability to Replicate AI Research (Starace et al., 2025) — the same OpenAI team's paper-replication benchmark.
- supports: OpenAI
- instance-of: AI Benchmarks and Evaluation