This is the paper describing HCAST, the task suite that supplies most of the tasks behind METR's time-horizon measurements. Its 22 authors are led by David Rein (corresponding) and Joel Becker, with Elizabeth Barnes as final author; most are at METR, with Chris Canal and Daniel O'Connell of Equistamp and two independent contributors. It was posted to arXiv as 2503.17354 on 21 March 2025, two days after METR's time-horizon paper (Measuring AI Ability to Complete Long Software Tasks), which drew on it.
Summary
The authors argue that forecasting the societal impact of autonomous AI requires benchmarks "with grounding, i.e., metrics that directly connect AI performance to real-world effects." HCAST's contribution is to measure how long each task takes skilled humans working under the same conditions as agents, so that agent success can be read as an answer to "can an agent be trusted to complete a task that would take a human X hours?"
Task construction
HCAST contains 189 tasks in 78 families sharing setup and scoring code; about 40% of families hold a single task and the rest three to four on average. By family, 30.8% are machine-learning engineering (training and fine-tuning, data science, debugging, scaffolding, data engineering), 20.5% software engineering (implementation, low-level optimisation, debugging, DevOps), 19.2% cybersecurity (capture-the-flag, cryptography, reverse engineering, a cryptocurrency exploit) and 29.5% general reasoning (common-sense, algorithmic, logical, statistical). METR staff and contractors created 143 tasks and 46 came through METR's task bounty for independent contributors. Each task is an instruction string, a containerised environment in which the agent runs as an unprivileged user, and a scoring function run as root; most scores are binary, and continuous scores are binarised at a heuristic success threshold that agents are not necessarily told. Tasks are chosen to be completable by writing and running code, to avoid heavy reliance on vision, and to "broadly play to models' strengths, while still being economically valuable."
Quality assurance had a human tester unfamiliar with each family attempt a task, gold reference solutions where feasible, and five agent runs per task (typically Claude 3.5 Sonnet (new)), with all 945 transcripts read manually to confirm that successes were legitimate rather than reward hacking and that failures were within the agent's control. On contamination, 125 tasks in 53 families have private solutions (86 with the problem itself private), 31 have public solutions the authors judge hard to memorise, and 33 have short public solutions that are easy to memorise. The paper carries a canary string and asks not to be included in training corpora.
Human baselines
Baseliners typically hold a degree from a top-100 university or have more than three years' relevant experience, are recruited mainly through METR staff networks, and pass a domain qualification task. They are paid $50–100 per hour plus $25–150 per hour in performance bonuses. They receive the same environment and instructions as agents, accessed over SSH, with web access allowed but not AI assistants, Wolfram Alpha or sign-in services; unlike agents they are told the success threshold. Some 140 people made 563 attempts on 139 of the 189 tasks, of which 341 succeeded; 110 tasks received at least one successful baseline. A task's time estimate is the geometric mean of successful baseline times, or a forecast from the task creator where none exists; forecasts correlate with baseline times at R² = 0.51. Estimated task lengths range from one minute to more than eight hours. Only 61% of baselines succeeded, which the authors caution is not comparable with agent success rates because incentives to move quickly encourage abandoning tasks; 79 tasks have no successful baseline (29 attempted without success, 50 not attempted).
Agent results
GPT-4o, o1, claude-3-5-sonnet-20241022 and Claude 3.7 Sonnet with extended thinking were run through METR's Modular scaffold (GPT-4o, Claude 3.5 Sonnet) and Triframe scaffold for reasoning models (o1, Claude 3.7 Sonnet), both ReAct-style, developed on a held-out set, with limits of 10 hours and 2 million or 8 million tokens respectively, at least five runs per task. Success fell consistently as estimated human time rose, with a marked drop between the 15-minute-to-1-hour bucket and the 1-to-4-hour bucket: the best models succeeded 70–80% of the time on tasks under one hour and under 20% on tasks over four hours. Claude 3.7 Sonnet and o1 led at every difficulty level. For roughly a third of tasks, successful agents took between 5 and 15 code-executing actions. The authors describe these results as "a reasonable lower bound," with better prompting and scaffolding likely to raise them.
Limitations stated by the authors
Tasks have limited, unambiguous scope and algorithmic scoring, unlike most real work; all are solitary software tasks without human interaction; economic relevance varies from close analogues of engineering work to toy reasoning exercises such as counting words in a file. Baseliners lack prior context, are imperfectly matched to their expertise, and were preferentially assigned harder tasks when they seemed strongest, which may bias time estimates. The benchmark is expensive and hard to maintain (GPU tasks, auxiliary machines, external APIs), limiting statistical power, and timing data contains residual errors.
Key claims
- 189 tasks, 78 families; 143 by METR, 46 by bounty; four domains; binary or thresholded scoring; canary string.
- 563 human baselines by about 140 people, over 1,500 hours, 341 successful (61%); 110 tasks with a successful baseline; task lengths one minute to 8+ hours; forecast–baseline R² = 0.51.
- Agents: 70–80% success under one human-hour, under 20% above four hours; Claude 3.7 Sonnet and o1 strongest.
- Contamination: 125 tasks with private solutions, 33 with easily memorised public solutions.
- Baseliner pay $50–100 per hour plus $25–150 per hour bonuses; web access allowed, AI assistants not.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); the abstract page (arxiv.org) records a single version submitted 21 March 2025, 32 pages, 10 figures and 5 tables. The HTML rendering of the abstract prints the four-hour figure as "44 hours" through a footnote artefact; the abstract page reads "more than 4 hours."
Relationships
- supports: METR Time Horizons — the source of the HCAST tasks in the time-horizon suite and of their human time estimates.
- depends-on: Measuring AI Ability to Complete Long Software Tasks — the companion time-horizon paper that uses HCAST alongside RE-Bench and SWAA.
- related: RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts (Wijk et al., 2024), Time Horizon 1.1 (METR, January 2026) — the latter added 73 further HCAST tasks to the suite.
- supports: METR
- instance-of: AI Benchmarks and Evaluation