"SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" is a paper from Scale AI introducing the SWE-Bench Pro benchmark. Its co-first authors are Xiang Deng and Jeff Da, with Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu and Brad Kenstler. It was posted to arXiv as 2509.16941 on 21 September 2025; the version summarized here is the second, of November 2025. Data are released on Hugging Face (ScaleAI/SWE-bench_Pro) and code on GitHub (scaleapi/SWE-bench_Pro-os).
Summary
Motivation
The authors state two limitations of existing issue-resolution benchmarks. The first is contamination: widely used repositories under permissive licences (MIT, Apache 2.0, BSD) "are prime candidates for inclusion in the large-scale web-crawled corpora used to pre-train LLMs." The second is that existing tasks understate real complexity; they count 161 of the 500 SWE-bench Verified tasks as requiring only one- or two-line changes, whereas enterprise work "often demands multi-file modifications spanning hundreds of lines." The benchmark is described as "explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-Bench," with tasks "that may require hours to days for a professional software engineer to complete."
Dataset design
SWE-Bench Pro holds 1,865 human-verified problems from 41 actively maintained repositories spanning consumer applications, business-to-business services and developer tools. It is split into a public set of 731 instances from 11 repositories under strong copyleft (GPL) licences, released openly; a held-out set of 858 instances from 12 further copyleft repositories, kept private to test for overfitting to the public set; and a commercial set of 276 instances from 18 proprietary codebases purchased from early-stage startups under partnership agreements, for which results but not problems are published. The authors describe the copyleft choice as creating "legal barriers" to inclusion in commercial training corpora and state that, to their knowledge, this is the first systematic use of the method for a research benchmark. Each repository contributes 50 to 100 instances, with a cap of 100. Trivial edits of 1–10 lines are excluded; reference solutions average 107.4 lines across 4.1 files, every problem changes at least 10 lines, and more than 100 tasks change more than 100 lines. Public tasks span Python, Go, JavaScript and TypeScript.
Problems are mined from pairs of consecutive commits, with the diff to test files taken as the test patch and the remainder as the gold patch. Each task description has three human-written parts: a problem statement rewritten in issue style from the original commit, pull-request and issue material; a list of requirements grounded in the tests, specifying expected behaviour without prescribing implementation; and, where relevant, an interface naming the classes and functions the tests expect, to avoid false negatives from valid solutions with different signatures. Professional engineers built Docker environments per repository; gold tests were run repeatedly to drop flaky problems; and each fail-to-pass test was human-checked for relevance and breadth, with problems dropped when no test survived. The authors note that real tasks can be underspecified and that the setting without requirements "is potentially interesting."
Results
All models were run through the SWE-Agent scaffold with its default prompt, a maximum of 50 turns, tool use where available, and model versions current as of 18 September 2025; Agentless was tried but "has difficulty in multi-file editing." Claude Opus 4.1 was run without extended thinking. Models received the problem statement, requirements and interface, so the benchmark measures implementation given a full specification rather than resolution of ambiguity. On the public set, pass@1 resolve rates were Claude Sonnet 4.5 43.6%, Claude Sonnet 4 42.7%, GPT-5 (high) 41.8%, Claude Haiku 4.5 39.5%, Kimi K2 Instruct 27.7% and GPT-OSS 120B 16.2%. On the commercial set, Claude Opus 4.1 scored 17.8%, GPT-5 (high) 15.7%, GPT-5 (medium) 14.9%, Gemini 2.5 Pro Preview 10.1%, Claude Sonnet 4 9.1% and GPT-4o 3.6%; the authors attribute the gap to "the difficulty of navigating enterprise codebases." The conclusion retains a statement that Opus 4.1 and GPT-5 achieve "a 23% success rate on SWE-Bench Pro compared to over 70% on benchmarks like SWE-Bench Verified."
Analysis
Analysis runs used a 50-turn and $2 cost cap. Go and Python tasks were resolved more often than JavaScript and TypeScript, whose rates ranged from near 0% to over 30% by model; some repositories stayed below 10% for all models while others exceeded 50% for some. Performance fell as the number of files rose, with Claude Opus 4.1 and GPT-5 staying above 10% on problems touching 10 or more files while open-weight models approached zero. Removing the requirements and interface cut GPT-5 (high) from 25.9% to 8.40% and Claude Opus 4.1 from 22.7% to 8.20%, which the authors read as unit-test false negatives when agents submit diverse valid solutions. A GPT-5 judge classified the last 20 turns of failed trajectories: for Opus 4.1, wrong solutions accounted for 35.9% and syntax errors 24.2% of failures; Claude Sonnet 4's leading failure was context overflow (35.6%) with 17.0% "endless file reading"; Gemini 2.5 split across tool errors (38.8%), syntax errors (30.5%) and wrong solutions (18.0%); Qwen3 32B had the highest tool-error rate (42.0%). Stated limitations are the uneven language mix, with Java, C++ and Rust underrepresented, and reliance on test suites that may reject valid alternative solutions.
Key claims
- 1,865 tasks from 41 repositories: 731 public (11 repositories), 858 held-out (12), 276 commercial (18 purchased startup codebases). (high)
- Reference patches average 107.4 lines across 4.1 files; edits of 1–10 lines excluded; over 100 tasks exceed 100 lines. (high)
- Public-set pass@1 with SWE-Agent, 50 turns, full specification: Claude Sonnet 4.5 43.6%, Claude Sonnet 4 42.7%, GPT-5 (high) 41.8%, Claude Haiku 4.5 39.5%, Kimi K2 Instruct 27.7%, GPT-OSS 120B 16.2%. (high)
- Commercial-set pass@1: Claude Opus 4.1 17.8%, GPT-5 (high) 15.7%, Gemini 2.5 Pro Preview 10.1%, GPT-4o 3.6%. (high)
- Without requirements and interface, GPT-5 (high) falls from 25.9% to 8.40% and Opus 4.1 from 22.7% to 8.20% under the capped analysis setting. (high)
- The authors count 161 of 500 SWE-bench Verified tasks as one- or two-line changes. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 2 (arxiv.org); the abstract page is arxiv.org. The project page is scale.com. The first arXiv version of 21 September 2025 reported lower scores under a unified scaffold; it was not separately retrieved.
Relationships
- supports: SWE-Bench Pro — the primary text for the benchmark's construction, splits and launch results.
- depends-on: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Introducing SWE-bench Verified (OpenAI, 2024) — the task format and the verification workflow it extends.
- depends-on: SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (Yang, Jimenez et al., 2024) — the evaluation scaffold and the failure-mode taxonomy method.
- supports: Why SWE-bench Verified no longer measures frontier coding capabilities (OpenAI, 2026) — the benchmark OpenAI adopted after retiring Verified.
- related: The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason (Liang, Garg and Zilouchian Moghaddam, 2025), AI Benchmarks and Evaluation, Scale AI