This is the paper by Samuel Miserendino, Michele Wang, Tejal Patwardhan and Johannes Heidecke of OpenAI that introduces the SWE-Lancer benchmark. Miserendino and Wang are marked as equal contributors. It was submitted to arXiv as 2502.12115 on 17 February 2025; the captured text is version 4 of 29 May 2025. The paper presents a set of real freelance software-engineering jobs whose difficulty is priced by the amounts a company actually paid for them, and reports that the frontier models it tested "are still unable to solve the majority of tasks."
Summary
Motivation
The authors argue that coding evaluations were saturating: when OpenAI announced SWE-Bench Verified in August 2024, GPT-4o scored 33%, and by the time of writing the o3 reasoning model reached 72%. They characterise SWE-Bench and SWE-Bench Verified as the most realistic prior benchmarks but say both draw on isolated, self-contained tasks from developer-facing open-source repositories and grade with unit tests, which they describe as "highly susceptible to grader hacking" and insufficient for full-stack work. SWE-Lancer is offered as a benchmark whose tasks carry real payouts, come from a user-facing product, span the full stack, and are graded by end-to-end tests written by professional engineers.
Dataset
SWE-Lancer consists of 1,488 freelance tasks posted on Upwork for the open-source repository of Expensify, which the paper describes as a $300 million public company (NASDAQ: EXFY) with about 12 million users. The tasks are collectively valued at $1,000,000 in payouts actually made to the freelancers who completed them, ranging from $250 to $32,000. Two task types are defined:
- Individual Contributor (IC) SWE tasks (764 tasks, $414,775): the model receives the issue text, the codebase checkpointed before the fix, and the objective of fixing the issue. Its patch is graded by end-to-end tests written in Playwright that simulate user flows in a browser; the model cannot see the tests.
- SWE Manager tasks (724 tasks, $585,225): the model receives the competing proposals that freelancers submitted on the original issue plus the codebase snapshot and must pick the best one. The selection is scored against the proposal the original engineering manager chose; a validation campaign with experienced engineers showed 99% agreement with those choices.
The construction pipeline had 100 professional software engineers review tasks, proposals and the codebase for clarity, specificity and executability; each IC task was reviewed by three engineers and each manager task by two, and IC tasks paying more than $5,000 were validated by a team of ten. Each end-to-end test was "triple-verified" and checked automatically to pass on the fixed code and fail on the broken code. An accompanying "user tool" lets the agent launch a scripted Playwright session against its own build and receive a text trajectory and screenshots, without being told whether the action succeeded.
A public split, SWE-Lancer Diamond, contains 502 tasks worth $500,800 (237 IC tasks worth $236,300 and 265 manager tasks worth $264,500); the remainder is a private holdout. In the Diamond set, 88% of IC tasks and 94% of manager tasks are bug fixes; 74% of IC tasks and 76% of manager tasks involve application logic and 17% and 18% involve UI/UX; 35% of tasks are worth more than $1,000 and 34% between $500 and $1,000. The average Diamond task took 26 days to resolve on GitHub and drew 47 comments, IC tasks required modifying on average 2 files and 69 lines, and manager tasks offer a choice among 4–5 proposals.
Experimental setup and results
Agents run in a Docker container without internet access, with the GitHub remote and later commits removed, and get a single attempt (pass@1) using a basic scaffold that can browse files, edit and run terminal commands. The baseline models were gpt-4o-2024-08-06, o1 at high reasoning effort, and Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), all with the user tool enabled; Claude 3.5 Sonnet was run on the full set at pass@1 only because of rate limits.
Claude 3.5 Sonnet performed best on both task types: 26.2% pass@1 on Diamond IC tasks (earning $58k of $236k) and 44.9% on Diamond manager tasks ($150k of $265k), for $208k of $501k on the whole Diamond set, and 33.7% overall on the full set earning $403k of $1M. o1 at high effort scored 16.5% on Diamond IC and 41.5% on Diamond manager ($166k on Diamond; $380k on the full set), and GPT-4o 8.0% and 37.0% ($139k on Diamond; $304k on the full set). Every model did better on manager tasks than on IC tasks; IC pass rates and earn rates stayed below 30%. Preliminary runs of two open-weight models, DeepSeek-R1 and Llama 3.3 70B Instruct, gave 3.8% and 4.6% on Diamond IC tasks and 40.4% each on Diamond manager tasks; the authors say these may be under-elicited.
Ablations show that more attempts and more reasoning both help. Allowing o1 six additional attempts (pass@7) nearly tripled the share of IC tasks solved, and GPT-4o at pass@6 matched o1 at pass@1 (16.5%). Raising o1's reasoning effort from low to high moved Diamond IC pass@1 from 9.3% to 16.5%, earnings from $16K to $29K, and earn rate from 6.8% to 12.1%, with gains concentrated on more expensive tasks. Removing the user tool reduced pass@1 only slightly (o1 from 16.5% to 13.1%), with stronger models losing more.
A price analysis over five runs on the 237 Diamond IC tasks estimates that using a model first and paying freelancers only for the tasks it fails would be 8.84% cheaper than freelancers alone with GPT-4o and 13.26% cheaper with o1 at one attempt, rising to 18.6% and 33.5% at five attempts; the "model contribution ratio" was $0.10 returned per $1 spent for GPT-4o and $0.16 for o1. The authors call this analysis preliminary and list its limits: it uses API prices including retries, the payouts are historical bounties, per-run cost varied (95% confidence intervals of $194.31 ± $17.31 for GPT-4o and $1,623.76 ± $56.68 for o1), and the five-attempt figure assumes quick verification, which "in practice can take over 24 hours."
Qualitatively, the authors report that agents localise the relevant code quickly through repository-wide keyword search but "fail to root cause," producing partial or flawed fixes, and that the strongest models make more effective use of the user tool despite its 90–120 second run time.
Limitations stated by the authors
The paper lists: tasks come from a single repository and a single freelance platform, so infrastructure engineering is underrepresented; freelance tasks are more self-contained than full-time engineering and all build on an established codebase, so the authors "remain cautious about extrapolating impact beyond this"; the evaluation is text-only although issue writers often attach screen recordings; models cannot ask clarifying questions; and Diamond tasks are public GitHub issues from 2023–2024, so contamination is possible and browsing should be disabled. A contamination check split Diamond tasks by model cutoff date and found no clear advantage on earlier tasks, for example 17.8% before versus 34.5% after the cutoff for Claude 3.5 Sonnet on IC tasks.
Key claims
- 1,488 tasks from the Expensify repository via Upwork, worth $1,000,000 in real payouts; 764 IC SWE tasks ($414,775) and 724 SWE Manager tasks ($585,225).
- IC tasks are graded by Playwright end-to-end tests, triple-verified by professional engineers; manager tasks by agreement with the original hiring manager's choice (99% expert agreement).
- SWE-Lancer Diamond, the public split: 502 tasks worth $500,800 (237 IC, $236,300; 265 manager, $264,500).
- Best result: Claude 3.5 Sonnet, 26.2% Diamond IC pass@1, 44.9% Diamond manager, $208,050 of $500,800 on Diamond and over $400,000 of $1,000,000 on the full set.
- o1 high effort: 16.5% Diamond IC, 41.5% Diamond manager; GPT-4o: 8.0% and 37.0%.
- Model-first with freelancer fallback: 8.84% (GPT-4o) and 13.26% (o1) cost savings at pass@1; 18.6% and 33.5% at pass@5.
- Stated purpose: to tie model performance to monetary value for research into the economic impact of AI development and into agentic safety.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 4 (arxiv.org); the abstract page is arxiv.org. The evaluation harness and Diamond split were released at github.com. The paper credits OpenAI's Preparedness, Safety Systems and Research organisations in its acknowledgements.
Relationships
- supports: SWE-Lancer — the primary text describing the benchmark's construction, grading and baseline results.
- related: SWE-bench — the prior benchmark the authors contrast with, on task source and unit-test grading.
- related: GDPval Paper (OpenAI, Oct 2025), GDPval — the later OpenAI evaluation of economically valued tasks that shares an author (Patwardhan).
- instance-of: AI Benchmarks and Evaluation
- related: AI Coding Agents, OpenAI