"The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason" is a paper by Shanchao Liang of Purdue University and Spandan Garg and Roshanak Zilouchian Moghaddam of Microsoft, first posted to arXiv as 2506.12286 in June 2025 and revised through version 4 on 1 December 2025. It presents diagnostic tasks designed to separate memorization from problem-solving on SWE-bench Verified and concludes that performance gains there "may be partially driven by memorization rather than genuine problem-solving."
Summary
Argument and method
The authors note that training corpora for large language models include the public GitHub repositories from which SWE-bench is built, and that models such as "Tools + Claude 4 Opus" had passed 70% pass@1 on the benchmark. Because memorization cannot be measured directly for commercial models, they propose a cross-benchmark comparison: a model with general coding skill should perform similarly on comparable tasks, while unusually high scores on one benchmark relative to similar tasks indicate memorization. Three diagnostic tasks are defined. In file-path identification, the model receives only the repository name and the issue text, with no code or repository structure, and must name the file the ground-truth patch modifies; a filtered accuracy excludes instances whose issue text already contains a file path or import. In function reproduction, the model receives the issue and a buggy file with the patched functions removed and must regenerate them without signatures or docstrings, scored by the share of generated 5-grams that appear in the ground truth. In prefix completion, the model receives the code preceding each modified hunk and is asked to continue; an instance counts as compromised if any hunk matches the ground truth exactly.
The comparison sets are SWE-bench Verified (500 tasks); 200 random instances of the full SWE-bench outside Verified; "SWE-bench Extra," 217 tasks from the same repositories drawn from issues created mostly after the original dataset's cutoff; 245 "Outside-Repo" tasks in SWE-bench style from seven popular repositories not in SWE-bench (jupyter/notebook, celery, aiohttp, scipy, numpy, pytorch, pandas); a 39-task subset of RefactorBench; and SWE-Bench-C#, an internal Microsoft benchmark of 75 tasks from eleven C# repositories. Verified and the C# set have similar repository sizes (763.5 versus 716.7 files) and issue lengths (451.2 versus 586.2 tokens); RefactorBench repositories are larger (1,149.2 files) with far shorter instructions (14.6 tokens). Ten models were tested: two GPT-4o snapshots, GPT-4.1, o3, o3-mini, o4-mini, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude Sonnet 4 and Claude Opus 4, all through provider APIs at default sampling.
Findings
On file-path identification, models score highest on Verified (60–76%), lower on the full SWE-bench sample (57–71%), lower again on the newer same-repository tasks (50–68%), and uniformly below 53% on the Outside-Repo tasks. The authors read the first gradient as instance-specific memorization of the widely circulated Verified problems and the second as repository-bias memorization of the twelve SWE-bench codebases. The ordering (Verified above Full above Extra above the external sets) holds for every model and vendor, which they take to indicate "systematic exposure patterns in training data" rather than a single provider's practice; the conclusion states the drop on external repositories reaches 47 percentage points. The patterns persist on the filtered instances, where 27.0% of Verified issues that mention a path have been removed. Among model families, OpenAI chat models showed no stable ranking across benchmarks, o3 scored highest, and Claude models improved by generation except Sonnet 4 on RefactorBench.
On function reproduction, 5-gram overlap with the ground truth reached 34.9% on Verified, 28.7% on the full set and 18.2% on SWE-bench Extra, the last matching the external benchmarks (RefactorBench 18.1%, Outside-Repo 13.9%) despite coming from the same repositories; the authors conclude that models have "specifically memorized the curated, canonical solutions" rather than general repository familiarity. Only o3-mini was tested among OpenAI reasoning models for this task. On prefix completion over Verified, the instance-level verbatim match rate ranged from 11.7% (o3-mini) to 31.6% (Claude 4 Opus); Claude models rose monotonically by generation (12.1%, 12.3%, 21.4%, 31.6%) while GPT-4.1 and the two GPT-4o snapshots sat at 17.4–18.4%. A contrastive measure, the difference between overlap with the patched and with the buggy code, was positive for Claude models on Verified, Full and Outside-Repo sets and smaller for OpenAI models, turned negative for all models except Claude 3.5 Sonnet and GPT-4.1 on the post-cutoff SWE-bench Extra split, and was largest for the GPT-4 family on the C# subset.
Conclusions and limitations
The authors call for benchmarks with temporal controls, cross-repository validation and cross-benchmark analysis, and describe their framework as reusable for any coding benchmark without access to training data or model internals. They acknowledge that the RefactorBench comparison is confounded by its much shorter issue descriptions, and that n-gram similarity is a noisy indicator because any correct fix shares some text with the reference.
Key claims
- File-path identification from issue text alone: up to 76% on SWE-bench Verified (o3), 57–71% on other full-SWE-bench instances, 50–68% on newer issues from the same repositories, under 53% on popular repositories outside SWE-bench. (high)
- Function reproduction 5-gram overlap: 34.9% Verified, 28.7% Full, 18.2% Extra, 18.1% RefactorBench, 13.9% Outside-Repo. (high)
- Prefix completion on Verified: 11.7% to 31.6% of instances reproduced a hunk verbatim; Claude 4 Opus 31.6%, Claude 4 Sonnet 21.4%, GPT-4o 18.2–18.4%, GPT-4.1 17.4%, Claude 3.5 Sonnet 12.1%, o3-mini 11.7%. (high)
- 27.0% of Verified issue descriptions contain a ground-truth file path; the memorization pattern persists after removing them. (high)
- The performance hierarchy is consistent across all ten models from both vendors. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 4 (arxiv.org, dated 1 December 2025); the abstract page is arxiv.org. The paper carries a CC BY-NC-ND 4.0 licence and ACM CCS classification "Software and its engineering, Maintaining software."
Relationships
- supports: SWE-bench — primary text for the memorization critique of the Verified subset.
- supports: AI Benchmarks and Evaluation — a method for detecting contamination through cross-benchmark comparison.
- supports: Why SWE-bench Verified no longer measures frontier coding capabilities (OpenAI, 2026) — OpenAI's later contamination findings are consistent with this paper's.
- depends-on: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Introducing SWE-bench Verified (OpenAI, 2024) — the datasets under examination.
- related: SWE-Bench Pro — a successor benchmark built with held-out and copyleft repositories to limit the exposure this paper measures.