This is the paper introducing the MCP-Atlas benchmark from Scale AI. It lists 23 authors, led by Chaithanya Bandi, Razvan-Gabriel Dumitru and Ben Hertzberg with Bing Liu as final author, affiliated to Scale AI and the National University of Singapore, with a group marked as equal contributors. It was posted to arXiv as 2602.00933 on 31 January 2026 and revised on 19 May 2026 (v3, the version summarised here), whose results reflect a May 2026 snapshot of server behaviour and a 20-model leaderboard.
Summary
The paper argues that existing evaluations of agents using the Model Context Protocol fall short on three axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic servers rather than mocks, and "structured, reproducible claim-level scoring disentangled from agent verbosity or style." MCP-Atlas targets three threats to validity the authors say prior work leaves uncontrolled: sanitised server behaviour that suppresses real error modes, trajectory-matching metrics that penalise valid alternative solutions, and clean tool sets that remove the need for tool discovery.
Benchmark design
The suite has 1,000 natural-language tasks written and verified by human experts over 36 production MCP servers exposing 220 tools in five categories: basic (30–35% of tasks; search, weather, maps), productivity (20–25%; filesystem, Notion, Slack, arXiv), coding (20–25%; git, GitHub, code executor, CLI), analytics (10–15%; Airtable, MongoDB) and financial (10–15%; Twelve Data, Alchemy). Servers run in containers with sandboxed filesystems and allow-listed egress, version-pinned and restarted between runs, but execute against live endpoints so that rate limits, pagination boundaries and transient errors are real. Tasks are split 500/500 into public and private subsets.
Each task exposes 6–37 tools (mean 15.2), of which 2–8 (mean 4.1) are required and the rest (mean 11.1) are distractors from semantically similar categories. Cross-server orchestration is required in 98.6% of tasks (mean 2.55 servers; 51.7% need exactly two, 38.2% three, 8.7% four or more). Prompts are single-turn and never name a server, tool or parameter. Each task carries a claims list of atomic, tool-grounded factual statements (mean 4.7, range 1–23); a judge scores each claim as fulfilled (1.0), partially fulfilled (0.5) or not fulfilled (0.0), task coverage is the mean, and a task passes at coverage ≥ 0.75. A reference trajectory (mean 9.8 tool-call steps) is used for solvability checks and diagnostics but never for scoring, so alternative valid trajectories receive full credit. Quality assurance has five layers: expert execution of the reference trajectory, domain-expert review, a second review for tool-name leakage, automated LLM verification of claim completeness, and random manual audit.
All models run under matched conditions — the same prompt, tool set, 100-call budget, rubric and answer protocol — with provider-native tool-calling adapters but no model-specific demonstrations, retries or scaffolding. Three judges score every task: Gemini 3.1 Pro Preview (primary), GPT-5.4 and Claude Opus 4.6. Failed tasks receive an automated diagnosis from an 11-category taxonomy split into tool-call issues (malformed parameters, wrong tool, no tool use, failed error recovery) and cognitive issues (task misunderstanding, faulty synthesis, response misparsing, early termination, hallucinated facts, logical errors, constraint violations).
Results
Across 20 models from Anthropic, Google, OpenAI, Meta, Kimi and GLM, the leaderboard shows three tiers. The top cluster, within overlapping confidence intervals, is Muse Spark 82.2%, Claude Opus 4.7 79.1% and Gemini 3.1 Pro Preview 78.2%. A mid-tier of eight models spans 67% to 77% (Opus 4.6, GLM-5.1, GPT-5.5, GPT-5.4, Gemini 3 Pro Preview, Claude Opus 4.5, Claude Sonnet 4.6, GPT-5.2); the remaining nine fall below 65%, down to Claude Haiku 4.5 at 40.2%. GLM-5.1 at 75.6% is described as the strongest open model, statistically tied with GPT-5.5 under the primary judge. Other figures given in the efficiency analysis include GLM-4.7 58.1%, o3 Pro 44.5%, GPT-5.1 50.1%, GPT-5.4 Mini 56.7% and GPT-5.4 70.6%.
Of roughly 6,900 diagnosed failures, 63.3% are cognitive and 36.7% tool-related. Tool failures dominate in weaker models: 40.1% of o3 Pro's failed trajectories contain no tool call at all, and GPT-5.1 and Claude Haiku 4.5 have tool-related shares of 50.4% and 46.7%. Among stronger models the bottleneck shifts downstream: Gemini 3.1 Pro Preview, GPT-5.4 and Claude Opus 4.6 terminate early on 42.8%, 41.4% and 36.3% of diagnosed failures, while GPT-5.5 and Claude Opus 4.7 nearly eliminate early termination (3.6% and 3.8%) but show faulty synthesis at 19.8% and 25.0%. On wall-clock time, six models sit on the Pareto frontier — Claude Haiku 4.5 (27 s, 40.2%), GLM-4.7 (42 s, 58.1%), Claude Opus 4.5 and Gemini 3 Pro Preview (about 50 s, 70%), Claude Opus 4.7 (71 s, 79.1%) and Muse Spark (121 s, 82.2%) — while o3 Pro (194 s, 44.5%) and GPT-5.4 Mini (135 s, slower than GPT-5.4 at 130 s) are dominated. The primary judge is the most generous (mean pass rate 65.4%), GPT-5.4 the strictest (62.3%), Claude Opus 4.6 intermediate (63.9%); per-model ranges across judges are 2.1–4.6 points, and Claude Opus 4.7 ranks second, third or fifth depending on judge, which the authors read as evaluator uncertainty rather than self-preference. Most models score higher on the public split than the private, with gaps from −0.8 to +9.6 points clustering by provider (OpenAI +8.4, Google +5.2, Anthropic +4.8, open-source +3.8); Kimi K2.5 is the only model scoring higher on the private split. The authors treat these gaps as audit signals, not evidence of contamination.
Limitations stated by the authors
All scoring is by language-model judges, so absolute pass rates carry a 2–5 point cross-judge band; servers and APIs are live and drift, so the leaderboard is a May 2026 snapshot; prompts are English-only; provider-recommended tool-calling strategies mean the harness is not identical across providers; the 100-call budget causes some failures on retry-heavy tasks; residual noise in claim lists bounds the achievable pass rate; and failure diagnostics may reflect the author-supplied evidence path where several paths exist.
Key claims
- 1,000 tasks, 36 production MCP servers, 220 tools, five categories; 500/500 public–private split.
- Per task: 6–37 exposed tools (mean 15.2), 2–8 required (mean 4.1), mean 11.1 distractors; 98.6% cross-server; mean 4.7 claims; reference trajectory mean 9.8 steps.
- Pass = mean claim score ≥ 0.75, judged by Gemini 3.1 Pro Preview with GPT-5.4 and Claude Opus 4.6 as secondary judges; 100-call budget.
- Top results (v3): Muse Spark 82.2%, Claude Opus 4.7 79.1%, Gemini 3.1 Pro Preview 78.2%; GLM-5.1 75.6%; Claude Haiku 4.5 40.2%.
- 63.3% of diagnosed failures cognitive, 36.7% tool-related; early termination is the leading single cognitive mode (18.7% overall).
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org); the abstract page (arxiv.org) records submission on 31 January 2026, revision on 19 May 2026, 25 pages, 3 figures and 9 tables. Earlier versions reported a different judge (Gemini 2.5 Pro), different per-task tool counts and a smaller model set; figures quoted from the January 2026 version are not directly comparable with the v3 leaderboard.
Relationships
- supports: MCP Atlas — the primary text for the benchmark's construction, scoring and results.
- supports: Scale AI
- depends-on: Model Context Protocol (MCP) — the interface the benchmark exercises.
- instance-of: AI Benchmarks and Evaluation
- related: Agentic AI, Agentic harnesses and capability elicitation