SWE-bench is a benchmark, introduced in a 2023 paper by Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan (Princeton), that evaluates whether language models can resolve real-world software issues by modifying a codebase. It was published as arXiv 2310.06770 and at ICLR 2024.
Summary
SWE-bench draws 2,294 tasks from GitHub issues and their merged pull-request fixes across 12 Python repositories, including django, scikit-learn, and matplotlib. Each task is framed end-to-end: given an issue and a snapshot of the repository, a model must produce a patch that passes the repository's own test suite. The authors state that resolving issues "frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously."
The benchmark was designed to be far from saturated at release. In 2023, Claude 2 solved 1.96% of issues.
Reception and later use
SWE-bench has since become a reference benchmark for Agentic AI coding capability, and a cleaned subset, SWE-bench Verified, is widely used as the reference metric. It is referenced in frontier-model system cards including GPT-5.3-Codex System Card, Claude Opus 4.6 System Card, and Claude Sonnet 4.6 System Card. Frontier agentic models in 2025–2026 routinely exceed 70% on SWE-bench Verified, consistent across those system cards. The movement from roughly 2% in 2023 to over 70% in 2025–2026 has been cited as evidence in discussions of coding-capability progress (Source: oneusefulthing.org) (Source: interconnects.ai). The results have been connected to METR's long-task scaling work and to the AI Software Progress concept.
Relationships
- supports: AI Benchmarks and Evaluation, Agentic AI, AI Software Progress
- related: Measuring AI Ability to Complete Long Software Tasks, (Source: oneusefulthing.org), (Source: interconnects.ai)
- instance-of: AI Benchmarks and Evaluation