"Why SWE-bench Verified no longer measures frontier coding capabilities" is an OpenAI research post published on 23 February 2026, credited to OpenAI as author, announcing that the company has stopped reporting scores on SWE-bench Verified and recommending that other developers report SWE-bench Pro instead. The post's subtitle states that "SWE-bench Verified is increasingly contaminated."
Summary
Argument
OpenAI writes that after the subset's August 2024 release it "provided a strong signal of capability progress and became a standard metric reported in frontier model releases," but that state-of-the-art progress had slowed, improving from 74.9% to 80.9% over the preceding six months according to a leaderboard the post cites. It asks whether remaining failures reflect model limits or the dataset, and reports two findings: tests that reject correct solutions, and training on solutions. It concludes that improvements on Verified "no longer reflect meaningful improvements in models' real-world software development abilities" but "increasingly reflect how much the model was exposed to the benchmark at training time." The post also states that OpenAI found evidence that models which have seen the problems in training are more likely to succeed, because they hold the extra information needed to pass underspecified tests.
Test audit
OpenAI audited 138 Verified problems, 27.6% of the set, that its o3 model did not consistently solve across 64 independent runs. Each was reviewed independently by at least six experienced software engineers, with any flagged issue re-verified by a further team. Of the 138, 59.4% had material issues in test design or problem description that made them "extremely difficult or impossible even for the most capable model or human to solve." Three categories are given: 35.5% with "narrow" tests that enforce specific implementation details and reject functionally correct submissions; 18.8% with "wide" tests that check functionality the problem description never specifies; and 5.1% with miscellaneous issues. Two examples are documented. In pylint-dev__pylint-4551, the tests import a function named get_annotation that the issue never mentions, so "many valid solutions fail the tests on import errors." In sympy__sympy-18199, the source pull request fixed three issues (#17373, #17377 and #18212) but the task description covers only #18212, so models that implement the described fix fail tests for the other two.
Contamination
The post says OpenAI first saw signs of contamination in its own models: GPT-5.2 solved 31 tasks the audit had identified as almost impossible, and in django__django-14725 its chain of thought showed knowledge of Django release notes, correctly identifying that the edit_only parameter the tests require was introduced in Django 4.1. To measure contamination more broadly, OpenAI built an automated red-teaming setup in which GPT-5, given each task's ID, description, gold patch and tests, spent 15 turns varying prompts, prefills and elicitation strategies against three non-reasoning targets, GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview, while a judge model rated how much novel task-specific information each response revealed, from "none" to "strong"; a second judge checked that the prober had not leaked the answer, and strong cases were reviewed by hand. The post states that all frontier models tested could reproduce the original gold patch or verbatim problem-statement specifics for some tasks. Three transcripts are reproduced: GPT-5.2 outputting the exact gold patch for django__django-11451, including the new early-return condition, from a fragment of the description; Claude Opus 4.5 recalling the file path, method, verbatim inline comment and four-line change for astropy__astropy-13236; and Gemini 3 Flash reproducing the task text and gold patch for django__django-11099, including the replacement regex, from the task ID alone.
Recommendations
OpenAI draws two lessons: benchmarks built from public material carry contamination risk, so developers should test for it, protect posted datasets (for example with passwords) and filter training data with canary strings; and automated scoring must be both agnostic to unimportant implementation details and robust to shortcut solutions, a problem the post calls "inherently complex" and which took "multiple extensive human labeling campaigns" to catch. OpenAI states that in recent months it has reported results on the public split of SWE-bench Pro and recommends others do the same; its contamination pipeline found some cases there, but "significantly rarer and less egregious," and no model produced a complete verbatim gold patch. It commits to investing in privately authored benchmarks, citing GDPVal as a model in which tasks are written privately by domain experts and graded by trained reviewers.
Key claims
- State-of-the-art SWE-bench Verified scores rose from 74.9% to 80.9% in the six months before the post. (high)
- Audit of 138 tasks (27.6% of the set) that o3 failed to solve consistently over 64 runs: 59.4% flawed, split 35.5% narrow tests, 18.8% wide tests, 5.1% other. (high)
- GPT-5.2 solved 31 tasks judged almost impossible, attributed to training exposure. (high)
- Under 15-turn automated probing by GPT-5, GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview each reproduced gold-patch or problem-statement content verbatim for some tasks. (high)
- OpenAI has stopped reporting Verified and recommends SWE-bench Pro's public split, where contamination was found but was rarer and never produced a complete gold patch. (high)
Provenance
Retrieved 22 September 2026 from openai.com. The post links the original SWE-bench paper (arXiv 2310.06770), the August 2024 Verified announcement, OpenAI's Preparedness Framework update, and GDPVal. The leaderboard figures are attributed in the post to llm-stats.com.
Relationships
- supports: SWE-bench — the primary text for the 2026 audit and contamination findings.
- supports: SWE-Bench Pro — OpenAI's stated reason for adopting SWE-bench Pro as its reported coding benchmark.
- supersedes: Introducing SWE-bench Verified (OpenAI, 2024) — revises the 2024 post's position that Verified is a reliable measure at frontier performance levels.
- supports: The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason (Liang, Garg and Zilouchian Moghaddam, 2025) — independent evidence of memorization on SWE-bench Verified.
- supports: AI Benchmarks and Evaluation — a documented case of contamination and test-validity failure in a widely reported benchmark.
- related: OpenAI, SWE-bench: Can Language Models Resolve Real-World GitHub Issues?