This is the paper introducing RE-Bench (Research Engineering Benchmark, V1) from METR. Hjalmar Wijk, Tao Lin and Joel Becker lead a 23-author list ending with Elizabeth Barnes; the alphabetically ordered middle authors include Holden Karnofsky (Anthropic; work done while at the Carnegie Endowment for International Peace), Josh Clymer (Redwood Research) and Nikola Jurkovic (Harvard). The paper is dated 20 November 2024, was posted to arXiv as 2411.15114 on 22 November 2024 and revised on 27 May 2025 (v2, summarised here). Environments are released at github.com/METR/ai-rd-tasks and agent trajectories at transcripts.metr.org.
Summary
The paper opens from frontier safety policies that name automation of AI research and development as a capability to anticipate, and observes that "there exist few evaluations for AI R&D capabilities, and none that are highly realistic and have a direct comparison to human performance." RE-Bench is built to three design goals for early-warning evaluations — feasibility, ecological validity and resistance to saturation — under two practical constraints: human experts are evaluated over eight hours, and every environment runs on eight or fewer H100 GPUs (at most six in practice). The stated logic runs one way: if agents do worse than human experts at these activities for the same cost, they are unlikely to automate those experts cost-effectively; the converse does not follow, because the environments are "very far from covering all the challenges involved in frontier AI R&D."
Environments and scoring
The seven hand-crafted environments each pose an ML optimisation problem: Optimize LLM Foundry (reduce a fine-tuning script's runtime without changing behaviour), Optimize a Kernel (a custom GPU prefix-sum kernel), Fix Embedding (recover a model whose embeddings were permuted), Scaling Law Experiment (predict the optimal hidden-size/steps trade-off for a 5e17-FLOP run using under 1e16 FLOPs of experiments), Restricted Architecture MLM (build a masked language model without division or exponentiation primitives), Finetune GPT-2 for QA (win rate judged by Llama-3 8B) and Scaffolding for Rust Codecontests (prompt and scaffold GPT-3.5 on 175 held-out problems). Each supplies a scoring function the agent can run at any time (except that Scaling Law Experiment hides the score), a simple starting solution and a withheld reference solution. Raw scores are linearly normalised so the starting solution is 0 and the reference solution 1; scores below the start count as 0, scores above 1 are possible, and none above 2 was observed. The best logged score counts, except in Scaling Law Experiment where only the final submission does.
Human experts
The 61 experts came from three pools: 43 applicants to a METR research scientist/engineer role who passed CV, CodeSignal, interview and task screens (45 runs, average normalised score 0.48); 11 people from METR staff networks with more than five years' relevant experience or recent work at Google DeepMind, Google, Anthropic, OpenAI, FAR Labs or Redwood Research (17 runs, 0.98); and 7 graduate students with relevant advisors or ML PhD programmes at Berkeley, CMU, Stanford or MIT (9 runs, 0.83). Overall, 71 runs averaged 0.64; 82% of attempts achieved a non-zero score and 24% matched or exceeded the reference solution. Because of the variance between pools, the authors "caution against narrowly focusing AI agent comparisons on the average human result." Validation found more than 70% of experts making progress in every environment by eight hours, more than 80% saying their score matched expectations, major technical issues in more than 5% of runs, and two environments (Finetune GPT-2 for QA and Scaling Law Experiment) whose top human solutions are likely within 15% of the maximum.
Agent results
Claude 3.5 Sonnet (claude-3-5-sonnet-20241022) and o1-preview were run in METR's Modular scaffold and in AIDE on Vivaria-provisioned machines (20–48 vCPUs, 200–400 GB RAM, 0–6 H100s), under wall-clock rather than token limits so that agents and humans get equal GPU hours, following MLE-bench. Over single eight-hour runs, agents made rapid early progress and then plateaued while humans started slowly and improved steadily. Under best-of-k with matched total budgets, Modular agents did best with many 30-minute attempts and AIDE agents with fewer two-hour attempts; the best AI agents scored 4× human experts at a two-hour budget, humans narrowly exceeded the top agent at eight hours, and humans reached twice the top agent's score at 32 hours. o1-preview in AIDE and Claude 3.5 Sonnet in Modular matched the 36th and 37th percentiles of human experts; with k = 128 thirty-minute samples agents remained far from top human performance. Results varied by environment: Claude 3.5 Sonnet approached human performance on Finetune GPT-2 for QA and Scaling Law Experiment, and o1-preview exceeded humans on Optimize a Kernel, where an agent wrote a faster custom Triton kernel than any human expert. Agents generated and tested solutions more than ten times faster than humans; an eight-hour run used about 29M input and 499K output tokens at roughly $123, against about $1,855 paid per human expert.
Discussion
The authors argue results may overstate automation capability because real AI R&D differs by at least two orders of magnitude on time horizon (8–32 hours versus six months or more), feedback loops (2 hours versus six months), engineering complexity (1,651 lines of code versus over a million) and parallel projects (1 versus over 100), and because tasks are cleanly defined and non-interacting. They also note ways results may understate it: agents are far cheaper, and AI may automate research through workflows unlike human ones. Other limitations include unrepresentative environments selected for clear scoring and fast progress, noisy results from rare high-scoring runs, cost, limited scaffold iteration, distance from siloed frontier research, possible overfitting to visible test scores, and luck in Scaling Law Experiment.
Key claims
- Seven environments; 71 eight-hour attempts by 61 experts; mean normalised score 0.64 (0.98 professional network, 0.83 graduate students, 0.48 hiring applicants); 82% non-zero, 24% at or above reference.
- Agents 4× humans at two hours; humans narrowly ahead at eight hours and 2× at 32 hours; best agents at the 36th–37th human percentile under best-of-k with an eight-hour budget.
- Agent eight-hour run about $123 (29M input, 499K output tokens) versus about $1,855 per human expert.
- Real AI R&D exceeds the benchmark by at least two orders of magnitude on four scale dimensions (Table 7).
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); the abstract page (arxiv.org) records submission on 22 November 2024 and revision on 27 May 2025.
Relationships
- supports: RE-Bench — the primary text for the environments, human baselines and launch results.
- supports: METR Time Horizons — RE-Bench environments form part of the time-horizon task suite (Measuring AI Ability to Complete Long Software Tasks).
- depends-on: MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (Chan et al., 2024) — adopts its AIDE scaffold and wall-clock budgeting.
- supports: METR
- related: HCAST: Human-Calibrated Autonomy Software Tasks (Rein et al., 2025), PaperBench: Evaluating AI's Ability to Replicate AI Research (Starace et al., 2025)
- instance-of: AI Benchmarks and Evaluation