Measuring AI Ability to Complete Long Software Tasks

"Measuring AI Ability to Complete Long Software Tasks" is a paper by Thomas Kwa, Ben West, Joel Becker, and colleagues at METR (Model Evaluation & Threat Research), published 2025-03-19 (Source: metr.org). It introduces a methodology for measuring the "time horizon" of generalist autonomous AI agents — the length of real-world software tasks, measured by how long human professionals take to complete them, that an agent can finish with 50% reliability. Its reported finding is that AI time horizons have been doubling roughly every 7 months, rising from minutes-long tasks to multi-hour tasks. The version in Raw Sources/ is the third arXiv revision (arXiv:2503.14499v3, February 2026), which includes o3 and o4-mini. Per-model estimates and later METR measurements are tabulated at METR Time Horizons.

Motivation

The authors argue that existing benchmarks consist of artificial rather than economically valuable tasks, are often adversarially selected against current models (HellaSwag and Humanity's Last Exam are the examples given), and saturate increasingly quickly, leaving no intuitive way to compare models of very different capability, such as GPT-2 and Claude 3.7. They frame capability tracking as a basis for governance, noting that many frontier developers have committed to using measures of specific capabilities to determine risk mitigations, and that sufficiently capable AI could perform dangerous actions such as autonomous development of CBRN weapons or self-replication outside human control. The idea of contextualizing AI capability by human time horizon is credited to Richard Ngo, with related concepts in work by Joseph Carlsmith and Ajeya Cotra.

Methodology

The 50%-task-completion time horizon is defined as "the time humans typically take to complete tasks that AI models can complete with 50% success rate."

Task suite

The paper assembles a suite of 170 diverse software-engineering and research tasks from three sources:

The results also include one GAIA task and five tasks requiring code robust to an adversary. SWAA was built to extend the trend back to pre-2023 models (GPT-2, davinci-002, gpt-3.5-turbo-instruct). Example tasks range from a 3-second multiple-choice question ("Which file is a shell script?") through a 56-minute data-munging script to an 8-hour task implementing custom CUDA kernels to speed up a backtesting tool by 30x. The authors describe one hour as roughly the shortest standalone economically relevant project and eight hours as a valuable software project. All tasks are scored automatically.

Human baselines

Both human professionals and AI agents attempt each task, and task difficulty is calibrated by how long a human professional takes. The authors used over 800 baselines totaling 2,529 hours: 558 baselines (286 successful) from HCAST and RE-Bench and 249 (236 successful) from SWAA. 148 of 169 tasks had human baselines; 21 HCAST tasks relied on researcher estimates. Baseliners were skilled professionals in software engineering, ML and cybersecurity, most from top-100 universities, with about 5 years of relevant experience on average. The time rating is the geometric mean of successful baseline times. The authors acknowledge that paying baseliners per success encouraged quick guesses and early abandonment, and that conditioning on success may make longer tasks' ratings underestimates; the measured human time horizon was around 1.5 hours rather than the 8 hours implied by the pay scheme, which they consider artificially low.

Models and fitting

The paper evaluates 12 frontier (and 4 near-frontier) models released between 2019 and 2025 with about 8 runs per agent/task pair, using METR's Modular and Triframe agent scaffolds; the most elicitation effort (2–3 engineer-weeks each) went to o1 and the original Claude 3.5 Sonnet. The metric tracks autonomous completion, with no human guidance during execution. Agent success on each task is binarized (continuous RE-Bench scores against the average of 7–9-hour human runs) and a logistic model adapted from Item Response Theory is fit per agent, with task difficulty set to log human time; the fitted 50% point is the time horizon. Tasks are weighted by the inverse square root of family size. Log time horizon is then regressed on release date, with 95% confidence intervals from a 10,000-sample three-level hierarchical bootstrap over task families, tasks and runs. Compute used included about 2,000 H100-hours for RE-Bench environments, 50,000 CPU-hours for other environments, and roughly 50,000 H100-hour equivalents for inference.

Findings

• Doubling time. Time horizon doubled every 207 days (95% CI 166–240 days) from 2019 to 2025. The 80% time horizon doubles on a similar schedule (204 days) but is 4–6x shorter, which the authors read as showing that even models that sometimes succeed on hard tasks "cannot reliably perform tasks of moderate length."
• Levels. GPT-2 has a 50% time horizon of about 2 seconds; o1 about 39 minutes; o3 about 110 minutes, with success on several tasks over 4 hours. As of the paper's original March 2025 publication, frontier model agents could complete tasks taking human experts roughly 1-2 hours. With continuous rather than binarized scoring, Claude 3.7 Sonnet's horizon is "nearly 2 hours."
• Possible acceleration. o3 lies above the long-run trend (p = 0.006). A 2023–2025 trend fitted without SWAA gives a 172-day doubling (about 20% faster than 207 days, within margin of error); 2024-only models suggest doubling about every three months; and 2024 models on the main suite give 143 days. The authors say their confidence in the 2024–2025 trend is low because only seven frontier models fall in that span.
• Success rates. Recent models complete about 50% of all tasks. Success falls with human task length (R² ≈ 0.80 against log human time), and the tasks models can complete are correlated across models (average correlation about 0.73). Average success rates by suite include o3 at 0.65 on HCAST and 0.40 on RE-Bench and Claude 3.7 Sonnet at 0.58 and 0.04.
• Drivers. Comparing 31 failed GPT-4 1106 runs with 32 failed o1 runs, the authors attribute progress to better tool use, adaptation to mistakes and logical reasoning: repeating failed actions accounted for 12 of GPT-4's failures but 2 of o1's, while premature task abandonment accounted for 16 of o1's. Agents still struggle without clear feedback loops (the "blackbox" and "symbolic regression" tasks) and rarely seek out information proactively.
• Cost. At a human rate of $143.61/hour (average Google L4 engineer salary divided by 2,000 hours), more than 80% of successful agent runs cost less than 10% of the human cost.

External validity

The authors ran three checks. On SWE-bench Verified, using its annotator time estimates, the exponential trend holds with a shorter doubling time of around 70 days, which they attribute partly to annotators underestimating easy-task times. Rating HCAST and RE-Bench tasks on 16 "messiness" factors (mean score 3.2/16, none above 8/16, against an estimated 9–15/16 for writing a good research paper), each additional point reduced success by roughly 8.1%, but improvement trends were similar across messier and cleaner subsets: on sub-hour tasks success rose about 40 percentage points between January 2023 and May 2025 in both. On five uncontaminated issues from METR's own repository, repository maintainers were 5–18x faster than contract baseliners, and agent performance matched contractor rather than maintainer time, suggesting time horizons correspond better to low-context human labor. The authors caution that time horizon is always relative to a domain, task distribution and baseliner skill, and that no tasks require a mouse, collaboration with other agents or very high reliability.

Extrapolation

The authors choose one month (about 167 working hours) as a threshold, citing Ngo's "1-month AGI" and the time new hires take to begin generating value, and state that they "cannot rule out" that a 1-month-horizon system could be transformative, "potentially including the ability for catastrophic harm." Naive extrapolation places 1-month horizons between mid-2028 and mid-2031 in the introduction; the extrapolation section gives an 80% CI about 2 years wide with a central estimate of mid-2029, and the discussion gives mid-2028 to mid-2030. If the 2024–2025 trend holds instead, half the probability falls in 2027 "and even late 2026." The authors note that forecasts are more sensitive to the doubling rate than to constant factors: a 2x change in doubling time shifts the 1-month date by 4.8 years, a 2x constant-factor error by only 0.6 years. They expect agency training and AI R&D automation to speed the trend (with substantial R&D automation likely once horizons reach tens of hours), compute limits to slow it, and argue that an AGI would have an "infinite" horizon, making the long-run trend faster than exponential.

Earlier summaries of the paper also describe an extrapolation to day-long tasks by late 2025 or early 2026 and week-long tasks by around 2027, and a finding that visual computer-use tasks have time horizons 40-100× shorter than coding tasks. Neither figure appears in the paper text in Raw Sources/; the computer-use comparison corresponds to later METR work on other domains.

Subsequent tracking

Later work has continued to apply and extend the time-horizon metric. Frontier models including Claude Opus 4.5 and GPT-5.2 have extended time horizons to roughly 5-6 hours at 50% reliability, consistent with the doubling trend (Source: epoch.ai) (Source: shumer.dev). Shumer cites METR data as quantitative evidence for the pace of AI capability growth and reports the doubling rate may be accelerating to roughly every 4 months (Source: shumer.dev). A self-evaluation of the AI 2027 forecast found METR coding time horizons tracking at 1.04× of its central trajectory (Source: blog.aifutures.org). Epoch AI's task testing in "How Close Is AI to Taking My Job?" directly references and extends METR's methodology (Source: epoch.ai). METR's running estimates, including later task-suite revisions, are recorded at METR Time Horizons.

Relationships

• instance-of: AI Progress Measurement
• supports: METR Time Horizons — the paper defining the metric and the original task suite
• depends-on: HCAST: Human-Calibrated Autonomy Software Tasks (Rein et al., 2025), RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (Wijk et al., 2024)
• related: Agentic AI — METR's time-horizon metric is a leading quantitative measure of agentic AI capability.
• related: Science of Scheming — METR's scaling curves for capabilities are the template Apollo Research has cited for replicating to scheming behaviors.
• related: METR, Grading AI 2027's 2025 Predictions (AI Futures Project, February 2026), SWE-bench

Provenance

PDF converted to markdown with images on 2026-04-13 (arXiv:2503.14499v3). Confidence stays medium: the page summarizes one paper, and the time-horizon figures are fast-decay capability claims.