"Introducing SWE-bench Verified" is an OpenAI research post, published on 13 August 2024 and updated on 24 February 2025, announcing a human-validated 500-task subset of the SWE-bench benchmark released jointly with the benchmark's Princeton authors. Its listed authors are Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn and Giulio Starace (equal contributors), with Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu and Aleksander Madry. The post frames the work as part of OpenAI's Preparedness Framework, under which autonomous completion of software-engineering tasks is a component of the Medium risk level in the Model Autonomy category.
Summary
Motivation
The post describes SWE-bench as a benchmark in which an agent receives a code repository and a GitHub issue and must produce a patch that makes the issue's FAIL_TO_PASS tests pass without breaking the PASS_TO_PASS tests; the tests are hidden from the agent. As of 5 August 2024 the leaderboard's top agents scored 20% on SWE-bench and 43% on SWE-bench Lite. OpenAI's testing identified tasks it judged hard or impossible to solve, so that the benchmark "systematically" underestimated capabilities, and it names three problems: unit tests that are overly specific or unrelated to the issue, underspecified issue descriptions, and development environments that are hard to set up reliably. The example given is scikit-learn__scikit-learn-14520, whose test requires a DeprecationWarning with an exact message that the issue text never mentions and that was settled only in pull-request discussion the agent cannot see. The post notes concurrent work by Xia, Deng, Dunn and Zhang on Agentless.
Annotation campaign
OpenAI worked with 93 software developers experienced in Python to screen 1,699 randomly chosen samples from the SWE-bench test set. Each sample was labelled by three annotators on two criteria, each on a 0–3 severity scale: whether the problem statement is underspecified (from "well-specified" at 0 to "almost impossible to understand" at 3) and whether the FAIL_TO_PASS tests could reject valid solutions (from "perfectly cover all possible solutions" at 0 to "too narrow/broad" at 3). Labels of 2 or 3 mark a sample as inadequate. Annotators also gave a free-form flag for other major issues and estimated how long an experienced engineer with a few hours of codebase familiarity would need for the fix, in four buckets: under 15 minutes, 15 minutes to 1 hour, 1–4 hours, and over 4 hours. The post cautions that these estimates "may overestimate the difficulty for a LLM, which may have memorized aspects of codebases and PRs." Prospective annotators had to pass onboarding tests built from 50 samples hand-labelled by OpenAI engineers. Severity labels were ensembled by taking the highest of the three annotators; difficulty by majority or, failing that, the median.
Results and dataset construction
Of the annotated samples, 38.3% were flagged for underspecified problem statements and 61.1% for tests that could unfairly reject valid solutions; 68.3% were filtered out overall, which the post says is "likely to be overzealous" but gives high confidence in what remains. Samples with an ensemble label of 2 or above on either criterion, or any flagged major issue, were removed, which is equivalent to removing any sample flagged by a single annotator. As many samples rated 1–4 hours and over 4 hours as possible were kept and the remainder sampled at random to reach 500. The final set contains 196 tasks rated as under-15-minute fixes and 45 rated as taking more than one hour. OpenAI states that Verified "supersedes the original SWE-bench and SWE-bench Lite test sets," released all annotations for slicing by difficulty, and, with the SWE-bench authors, built a containerized Docker evaluation harness.
Performance
Running gpt-4o-2024-05-13 through several open-source scaffolds (Agentless, AutoCodeRover, Moatless Tools, Aider and SWE-Agent) with a single seed and default or closest-documented hyperparameters, GPT-4o reached 33.2% on Verified with the best scaffold, Agentless, against 16% on the original SWE-bench. The gain from SWE-bench Lite was smaller because Lite was already filtered toward easier tasks. Stratifying by difficulty, the post reports that resolution rates rose within individual difficulty buckets when moving to Verified, which it reads as consistent with removing impossible samples rather than merely shifting toward easy ones; the effect is clearest in the two easiest buckets, where samples are most numerous.
Discussion and limitations
The post draws three lessons: to invest in understanding benchmarks deeply, since increasingly hard tasks demand more expertise to curate; to account for ecosystem progress, noting that GPT-4's SWE-bench Lite score ranged from 2.7% with an early RAG scaffold to 28.3% with CodeR, so evaluations should be run continually including after training; and to recognise the limits of static datasets, since the tasks are scraped from public GitHub repositories and "large foundation models that are pre-trained on internet text are likely to be contaminated." SWE-bench is described as covering only a narrow slice of the Medium autonomy risk level and requiring supplementary evaluations.
Key claims
- 93 Python developers annotated 1,699 SWE-bench test samples, three labels each, ensembled by highest severity. (high)
- 38.3% flagged for underspecification, 61.1% for unfair tests, 68.3% filtered out overall. (high)
- SWE-bench Verified contains 500 tasks: 196 under-15-minute fixes, 45 over one hour. (high)
- GPT-4o (May 2024 checkpoint): 33.2% on Verified with Agentless, up from 16% on the original SWE-bench. (high)
- Leaderboard as of 5 August 2024: 20% on SWE-bench, 43% on SWE-bench Lite. (high)
- GPT-4 on SWE-bench Lite: 2.7% (early RAG scaffold) to 28.3% (CodeR), showing scaffold dependence. (high)
Provenance
Retrieved 22 September 2026 from openai.com. The dataset is hosted at huggingface.co/datasets/princeton-nlp/SWE-bench_Verified; the annotation results and rubric are linked from the post as downloads from cdn.openai.com. The post's acknowledgements credit Tao Lin with initially pointing out many of the issues.
Relationships
- supports: SWE-bench — the primary text for the Verified subset's construction, difficulty labels and launch score.
- depends-on: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — the original benchmark it filters.
- supersedes: the original SWE-bench and SWE-bench Lite test sets, per the post's own statement (no separate page).
- contradicts: Why SWE-bench Verified no longer measures frontier coding capabilities (OpenAI, 2026) — OpenAI's later finding that residual test flaws and contamination made Verified unsuitable for frontier reporting; the later post revises, rather than disputes, this one.
- supports: Benchmark Performance Over Time, AI Benchmarks and Evaluation
- related: OpenAI