"Remote Labor Index: Measuring AI Automation of Remote Work" is a benchmark paper from the Center for AI Safety and Scale AI, posted to arXiv (2510.26787) on October 30, 2025. It has a large author list; the equal-contribution first authors are Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal and Yury Orlovskiy, and the senior authors are Summer Yue, Alexandr Wang, Bing Liu, Ernesto Hernandez and Dan Hendrycks. The benchmark it introduces is covered in its own right at Remote Labor Index.
The measurement gap
The paper's premise is a disconnect between benchmark progress and economic effect: "AIs have made rapid progress on research-oriented benchmarks of knowledge and reasoning, but it remains unclear how these gains translate into economic value and automation."
It names the resulting problem for policy: "we lack standardized, empirical methods for monitoring the trajectory of AI automation." The authors argue that without "reliable metrics grounded in real-world economic activity, stakeholders may struggle to build consensus and proactively navigate AI-driven labor automation," and that existing agent benchmarks, which measure specialized skills such as software engineering and basic computer use or simple tasks shared across professions, "offer limited insight into the trajectory of human labor automation." They distinguish RLI from OpenAI's GDPval work (Patwardhan et al.), which they describe as showing models near human parity on specific tasks shared across many professions, indicating potential for augmentation without measuring "the capacity for end-to-end project automation" (GDPval Paper (OpenAI, Oct 2025)).
The benchmark
The Remote Labor Index (RLI) is "a broadly multi-sector benchmark comprising real-world, economically valuable projects designed to evaluate end-to-end agent performance in practical settings." The authors describe it as the first standardized, empirical measurement of AI's capability to automate remote work.
Two design choices distinguish it from capability benchmarks. The tasks are real projects with economic value rather than constructed problems, and evaluation is end-to-end — the agent must deliver a complete piece of work rather than demonstrate a component skill.
Composition
Each of the 240 projects has three parts: a text brief, a directory of input files, and a gold-standard human deliverable, all supplied by the professional who did the work, together with that professional's reported time and cost. Example projects shown in the paper include an interactive dashboard for World Happiness Report data, 3D product animations for an earbuds design, a 2D animated advertisement with a supplied voice-over, architectural plans and a 3D model for a container home, a physics-based browser game under 5 MB, and formatting a paper for an IEEE conference.
The projects span 23 of the 64 categories in the Upwork taxonomy, which the authors chose over O*NET as better suited to remote freelance work. The largest categories are video (31%), CAD (12%), graphic design (11%), game development (10%), audio (10%), architecture (7%) and product design (6%), with 13% other. The authors contrast this with earlier benchmarks concentrated on software, research and writing tasks, where current models already do relatively well.
Human completion time averages 28.9 hours (median 11.5; range 0.2 to 450 hours), which the authors say matches a random sample of Upwork jobs and is more than twice that of earlier benchmarks such as HCAST and GDPval (HCAST: Human-Calibrated Autonomy Software Tasks (Rein et al., 2025)). Project cost averages $632.60 (median $200; range $9 to $22,500). In total the projects represent over 6,000 hours of work valued at over $140,000; completing every project would earn $143,991.
Collection
Collection started from Upwork's 64 categories, filtered to 43 eligible ones by excluding physical work (such as local photography), work whose results take time to evaluate (such as SEO), and work that cannot be evaluated in a web-based platform (such as back-end development). A job post in each eligible category yielded 207 projects; long-tail sourcing yielded 7 more from categories outside the taxonomy and commissioned work, and 33 from high-quality work found online whose authors gave permission and supplied time and price information. The authors recruited 358 freelancers with verified Upwork accounts, averaging 2,341 hours worked, 89 prior jobs and $23,364 in Upwork earnings, and paid them between $15 and $200 per project (average $41) for existing work samples. From 550 initial projects, multiple review rounds excluded projects that required human interaction or produced proprietary formats that could not be rendered, leaving 240.
To protect personal information and limit contamination, 230 projects form a private test set and 10 are public, released with open-source evaluation-platform code; the authors state that none of the project descriptions are searchable and that a domain blocklist covers long-tail deliverables that exist online.
Metrics and evaluation
The paper defines four metrics:
- Automation rate — the share of projects where the AI deliverable is judged to complete the project at least as well as the human deliverable.
- Elo — relative performance between agents from pairwise comparisons, with the human baseline fixed at 1,000; a 400-point difference corresponds to 10:1 odds.
- Dollars earned — the summed human cost of the projects the agent completes.
- Autoflation — the percentage fall in the cost of completing the full project bundle when each project is done by the cheapest successful method, human or AI.
Because judging deliverables "is itself a highly agentic task," the authors state that automated LLM evaluation is not feasible and all evaluation is manual. Evaluators take the perspective of "a reasonable client" and rate each AI deliverable against the human reference on a three-point scale (would not be accepted; would be accepted; would be accepted and exceeds the reference). The authors abandoned granular per-project rubrics after finding that a deliverable could satisfy rubric items yet fail professional standards. Three independent evaluations are combined by majority vote. Inter-annotator agreement was 94.4% for the automation rate; for Elo comparisons, ternary agreement was 56.9% against 33.0% chance, with hard disagreements at 5.9%. Evaluations averaged 11.4 minutes (automation rate) and 17.4 minutes (Elo) against caps of 20 and 30 minutes. The authors expect agreement on the automation rate to fall as AI deliverables become more complex.
The result
"AI agents perform near the floor on RLI, with the highest-performing agent achieving an automation rate of 2.5%."
| Agent | Automation rate |
|---|---|
| Manus | 2.5% |
| Grok 4 | 2.1% |
| Claude Sonnet 4.5 | 2.1% |
| GPT-5 | 1.7% |
| ChatGPT agent | 1.3% |
| Gemini 2.5 Pro | 0.8% |
Models supporting computer use ran in a computer-use scaffold developed by Scale AI; others ran in the OpenHands command-line scaffold. GPT-5 was run in both and reported with the command-line scaffold, which scored higher. Dollars earned and autoflation were likewise near the floor. On Elo, all agents fell short of the human baseline of 1,000, but the rankings generally placed newer frontier models above older ones, which the authors take as evidence that the benchmark is "sensitive enough to detect ongoing progress."
The gap between this figure and performance on research benchmarks is the paper's substantive finding: the same systems that score highly on knowledge and reasoning evaluations complete very few real projects end to end. The conclusion describes "a stark gap between progress on computer use evaluations and the ability to perform real and economically valuable work." The stated purpose is to "ground discussions of AI automation in empirical evidence, setting a common basis for tracking AI impacts."
Failure modes and successes
Clustering evaluator justifications from roughly 400 evaluations, the authors report the share of AI deliverables with each class of problem (categories overlap): poor quality 45.6%, incomplete deliverables 35.7%, corrupted or empty files 17.6%, and inconsistencies between files 14.8%. Examples include videos of 8 seconds where 8 minutes were requested, child-like drawings built from basic shapes, a house whose appearance changes between 3D views, robotic voice-overs, floor plans that do not match supplied sketches, and working web games with graphics below professional standard. Successes were concentrated in audio work (sound effects for a retro game, separating vocals from accompaniment, merging voice-overs with music), image generation (ads and logos), report writing and code for interactive data visualizations; the paper shows a Sonnet 4.5 World Happiness dashboard as a successful completion. Drawing on Hendrycks et al.'s definition of AGI, the authors attribute many failures to agents' inability to verify and fix their own work, especially in projects requiring interactive audiovisual checking such as architecture, game development and web development.
Discussion and limitations
The authors argue that AI differs from earlier, task-specific automation technologies such as the electronic calculator because it is being developed "to automate human intelligence itself," and that an AI that automated all current remote work without overfitting would likely have general cognitive skills allowing it to automate new jobs as they arise.
They list as limitations that RLI excludes work requiring client interaction (such as tutoring) or teamwork (such as project management), so that 100% on RLI would not imply human-level performance on all remote work; and that project costs are not adjusted for inflation and, for projects mostly completed within the previous five years, likely understate current economic value.
Bearing on the labor debate
The 2.5% figure is a measurement of end-to-end automation at the time of the paper, not a forecast, and the authors present the Elo metric as the means of tracking change over time. It supplies a direct empirical answer to the question Newman treats as unresolved, and sits alongside the Stanford early-career finding, which the paper cites (Brynjolfsson, Chandar and Chen, "Canaries in the Coal Mine?"), as evidence of the gap between demonstrated capability and realized displacement. It bears on the near-term phase of the staged frameworks in Anthropic's Economic Policy Framework (June 2026) and A Roadmap for the Upcoming Labor Transition (Cheng & Schaal, June 2026), both of which assume displacement is not yet broadly visible. A February 2026 Epoch AI comparison of RLI with APEX-Agents and GDPval judged that RLI's early evaluations "likely under-elicited models" and recorded a top RLI score of 4% at that time (What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026)).
Provenance
Captured from the arXiv PDF (arxiv.org; abstract page arxiv.org), main text pages 1–12. Appendix material on per-environment results and additional examples is not summarized here.
Relationships
- supports: AI Labor Disruption — supplies a direct end-to-end automation measurement against capability-benchmark inference
- supports: Anecdotes Everywhere, Evidence Almost Nowhere (Steve Newman, July 2026) — independent evidence for the claim that measurable displacement remains small
- related: AI Benchmarks and Evaluation — argues research benchmarks do not measure economic substitution
- related: Remote Labor Index — the benchmark's own page, with results over time
- related: GDPval Paper (OpenAI, Oct 2025), HCAST: Human-Calibrated Autonomy Software Tasks (Rein et al., 2025), What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026), A Definition of AGI (Hendrycks, Song, Szegedy et al., 2025)
- related: AI and Productivity, Agentic AI, Dan Hendrycks, Scale AI