This is the paper by Bertie Vidgen, Austin Mann, Abby Fennelly and 21 co-authors at Mercor, a talent marketplace, that introduces the AI Productivity Index for Agents (APEX-Agents), a benchmark of professional-services tasks for AI agents. It was submitted to arXiv as 2601.14242 on 20 January 2026; the captured text is version 3 of 23 February 2026. The dataset of 480 tasks is released on Hugging Face under a CC-BY licence together with Archipelago, Mercor's open-source infrastructure for running and grading agents.
Summary
Motivation and survey
The authors argue that existing agentic evaluations have "a large sim-to-real gap": they are narrowly scoped, contrived, and made of simple tasks that do not capture how professionals work. To ground the benchmark they surveyed 227 experts from the Mercor marketplace (58 financial and investment analysts, 77 management consultants, 92 lawyers; 10.8 years of experience on average) about how they spend their time. Core activities accounted for 47% of working time, and free-text descriptions of those activities were grouped into 18 categories that informed task creation.
Worlds and tasks
The benchmark was built in three steps. First, professionals created 33 data-rich "worlds", each based on a project scenario (10 in investment banking, 11 in management consulting, 12 in law). Teams were assigned in-world roles such as partner or associate and delivered the project over 5–10 days, exchanging emails and messages, conducting research and producing spreadsheets, reports and slide decks; one expert played the customer. Worlds contain 166 files on average. Of the 33, 22 involve entirely fictional entities, 9 place real companies in fictional scenarios and 2 mix both. Each world exposes nine applications comprising 63 tools (calendar, chat, code execution, documents, file system, mail, PDFs, spreadsheets, presentations); two investment-banking worlds add EDGAR SEC and equity and fixed-income market data applications with a further 187 tools. Web search is disabled for reproducibility.
Second, experts wrote 480 tasks (160 per job; 8–20 per world, 14.5 on average) as single-turn prompts that can only be completed with the in-world files and applications. Contributors could run frontier agents (GPT-5, Claude Opus 4.5, Gemini 3 Pro) while writing, "allowing them to adversarially iterate." Of the 480 tasks, 422 require a message in the console and 58 require creating or editing a spreadsheet, document or presentation. An example asks for the P/E ratio for KVUE using the implied share price in a DCF model and diluted EPS from dated financials. Experts estimated that tasks take professionals 1.82 hours on average (1.36 for investment banking, 2.40 for law, 1.69 for consulting); the introduction separately states an average estimate of 11–22 hours.
Third, experts wrote rubrics of short true-or-false criteria (1 to 10 per task, mean 4.06) that "only reward critical aspects of the output," plus a gold output per task, and tagged each with output type, workflow and expected time. In total 256 experts contributed (mean 12.9 years of experience, median 11.0), including former BCG and McKinsey consultants, Morgan Stanley and Citigroup bankers, and corporate lawyers.
In a baselining study, experts who had not seen a task independently executed 20% of them (n=96). Minor problems were found in 10% of these and fixes cascaded to the dataset. Experts had estimated 1.70 hours for the sampled tasks; the measured time was 1.37 hours, an over-estimate of 33%.
Evaluation method
Eight agents were run eight times on every task, producing 30,720 trajectories, with a cap of 250 steps after which a run is marked a failure. All agents use a ReAct toolbelt and a lightly optimised system prompt; closed models were called through their APIs and open-weight models through Baseten. Gemini 3 Flash with thinking set to low grades each criterion independently from the prompt, the agent's output and a change log, without seeing the trajectory. Against 747 human labels on 249 criteria, the judge reached 98.5% accuracy, 96.7% precision and 98.1% recall (false negative rate 1.9%, false positive rate 1.3%), so the authors advise caution over score differences below one percentage point. Because the judge is also on the leaderboard, the authors checked its 84 own-criteria labels and found one false positive (1.2%), in line with other agents.
The leaderboard metric is Pass@1, the task-uniform mean of per-task pass rates where a pass requires meeting every criterion, with 95% confidence intervals from 10,000 task-level bootstrap resamples. Pass@8 (at least one success in eight runs), Pass^8 (success in all eight) and the mean share of criteria met are reported alongside.
Results
Gemini 3 Flash scored highest at 24.0% Pass@1 [20.7–27.3], followed by GPT-5.2 at 23.0% [19.8–26.2], Claude Opus 4.5 and Gemini 3 Pro at 18.4% each, GPT-5 at 18.3%, Grok 4 at 15.2%, GPT-OSS-120B at 4.7% and Kimi K2 Thinking at 4.0%. Where available, thinking or reasoning effort was set to high. Under McNemar's exact tests with a Benjamini–Hochberg correction, the gap between Gemini 3 Flash and GPT-5.2 is not significant, but both are significantly better than all other agents. By job, the best Pass@1 was 27.3% for investment banking (GPT-5 and GPT-5.2), 22.7% for consulting (GPT-5.2) and 25.9% for law (Gemini 3 Flash).
Pass@8 rose to 40.0% for GPT-5.2, 37.3% for Gemini 3 Pro and 36.7% for Gemini 3 Flash, about 15 points above Pass@1, which the authors read as agents having capability but being inconsistent; Pass^8 peaked at 13.4% (Gemini 3 Flash). Mean criteria met was 39.5% for Gemini 3 Flash, 38.7% for GPT-5.2 and 34.8% for Claude Opus 4.5. Every agent scored zero on at least 40% of runs. Gemini 3 Flash used nearly five times as many tokens per run as GPT-5.2 (5.3 million against 1.0 million) and about eight times as many as Gemini 3 Pro; Kimi K2 Thinking averaged 92 steps and 91 tool calls per task and timed out on 29.8% of trajectories. Agents scored lower on the 58 tasks requiring a file output. Unrequested file deletions occurred in 36 trajectories (0.12%), 21 of them by GPT-5.2. In head-to-head comparisons of passing and failing runs on the same task, passing runs took 5.95 fewer steps and 5.66 fewer tool calls, which the authors attribute to avoiding "doom looping."
The conclusion states that the top agents score under 25% on Pass@1 and no more than 40% on Pass@8 or mean score, and that future versions will expand task horizon, world complexity and represented value.
Key claims
- 480 tasks across 33 worlds, 160 per job; 256 expert contributors; 422 console-output and 58 file-output tasks; mean 4.06 rubric criteria per task.
- Leaderboard Pass@1 (thinking high, 8 runs): Gemini 3 Flash 24.0%, GPT-5.2 23.0%, Claude Opus 4.5 18.4%, Gemini 3 Pro 18.4%, GPT-5 18.3%, Grok 4 15.2%, GPT-OSS-120B 4.7%, Kimi K2 Thinking 4.0%.
- Judge (Gemini 3 Flash, thinking low) accuracy 98.5% against 747 human labels; differences under one point should be read cautiously.
- Expert time estimates over-state measured completion time by 33% (1.70 versus 1.37 hours) on the 96 baselined tasks.
- No human pass rate is reported; web search is off; the 250-step cap marks a run as failed.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); the abstract page is arxiv.org. Dataset: huggingface.co. Infrastructure: github.com. Corresponding address apex@mercor.com.
Relationships
- supports: APEX-Agents — the primary text for the version 1.0 benchmark design and baseline scores.
- related: GDPval, Remote Labor Index — the other economically framed agent benchmarks that What do 'economic value' benchmarks tell us? (Brand and Burnham, Epoch AI, 2026) compares it with.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, Gemini 3 / Gemini 3 Pro, GPT-5 Family (OpenAI), Claude Opus 4.5