"SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering" is a paper by John Yang and Carlos E. Jimenez (equal contributors), Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan and Ofir Press of Princeton Language and Intelligence at Princeton University, with correspondence addresses at Stanford and Princeton. It was posted to arXiv as 2405.15793 in May 2024 and revised to version 3 on 11 November 2024. The paper introduces SWE-agent, an open-source system in which a language model solves SWE-bench tasks by interacting with a computer, and the concept of the agent-computer interface (ACI), the layer of commands and feedback formats through which the model acts. Data, code and a leaderboard are at swe-agent.com.
Summary
Argument
The authors posit that language-model agents "represent a new category of end users with their own needs and abilities" and would benefit from purpose-built interfaces, in the way human engineers benefit from integrated development environments. Agents operating a raw Linux shell, they report, struggle to edit small segments of a file reliably and receive no feedback on invalid edits. Drawing on human-computer interaction methods, they iterated on an ACI by inspecting agent behaviour on a development set and grid-searching configurations, arriving at four design principles: actions should be simple with few options; actions should be compact so that navigation and editing take one step rather than several; environment feedback should be informative but concise; and guardrails such as a syntax checker should limit error propagation. Unlike human interfaces, the paper notes, every token of context has a fixed cost for a model, so distracting content harms performance.
System design
SWE-agent runs on the ReAct pattern, generating a thought and a command at each step, on top of a Linux shell whose ordinary utilities remain available. Its components are: search and navigation commands (find_file, search_file, search_dir) that return summarized results capped at 50 hits, above which the agent is told to refine its query; a file viewer (open, scroll_up, scroll_down, goto) that shows a window of at most 100 numbered lines together with the file path, total length and lines omitted; an edit command that replaces a line range in the open file in a single step and redisplays the result, with an integrated linter that discards edits introducing syntax errors and asks the agent to retry; and context management that collapses every observation before the last five to a single line, reports empty command output explicitly, and drops all but the first of any repeated malformed-generation errors.
Experiments and results
Evaluation uses the full SWE-bench test set of 2,294 tasks from 12 Python repositories and the 300-task SWE-bench Lite subset, plus HumanEvalFix for short-form debugging, with GPT-4 Turbo (gpt-4-1106-preview) and Claude 3 Opus (claude-3-opus-20240229); Llama 3 and DeepSeek Coder were tried and found weak in the agent setting, Llama 3's 8k context being too small. A per-instance budget of $4 was imposed, with existing edits submitted when exceeded. Baselines are the non-interactive BM25 retrieval-augmented generation setting of the SWE-bench paper and a "Shell-only" agent adapted from InterCode.
SWE-agent with GPT-4 Turbo resolved 12.47% of the full test set (286 of 2,294) at an average cost of $1.59 per resolved instance, and 18.00% of Lite (54 of 300); with Claude 3 Opus, 10.46% and 13.00%. RAG baselines scored 1.31% (GPT-4 Turbo) and 3.79% (Claude 3 Opus) on the full set, and the Shell-only agent 11.00% on Lite (7.33% without a demonstration). The paper describes this as a 64% relative gain over Shell-only and a 6.7-fold gain over RAG at 8–13 times the cost, against a prior best of 3.8% for a non-interactive system. On HumanEvalFix, SWE-agent with GPT-4 Turbo reached 87.7% pass@1 in Python, 89.7% in JavaScript and 87.9% in Java, against 47.0–50.0% for GPT-4 without the agent as reported by prior work (the results section separately states 88.3%).
Ablations on Lite, each relative to the 18.0% full configuration: removing the linter, 15.0%; removing the edit command, 10.3%; replacing summarized search with iterative one-result-at-a-time search, 12.0%, worse than having no search tool (15.7%); a 30-line viewer window, 14.3%; showing the full file, 12.7%; keeping full history, 15.0%; no demonstration, 16.3%. The authors observe that agents given iterative search inspect every match exhaustively and exhaust their budget.
Agent behaviour
Trajectories typically begin by writing a reproduction script or localizing the issue, then settle into edit-then-execute loops. Editing remains difficult: 51.7% of GPT-4 Turbo trajectories on the full set (1,185 of 2,294) contain at least one failed edit, and although 90.5% of edit attempts eventually succeed, that probability falls to 57.2% after a single failure. Successful runs finish earlier and cheaper, with a median of $1.21 and 12 steps against a mean of $2.52 and 21 steps for failures, and 93.0% of resolved instances are submitted before the budget is exhausted against 69.0% overall, which leads the authors to expect little gain from larger budgets. A GPT-4o classifier, agreeing with the authors on 87% of a validation set, attributed 52.0% of the 248 unresolved Lite instances to incorrect or overly specific implementations and 23.4% to cascading failed edits. Appendix analyses report that success is uncorrelated with issue age, and that per-instance outcomes vary considerably across runs even though average performance is stable.
Key claims
- SWE-agent with GPT-4 Turbo: 12.47% (286/2,294) on SWE-bench, 18.00% (54/300) on Lite, $1.59 average cost per resolved instance; with Claude 3 Opus, 10.46% and 13.00%. (high)
- RAG baselines: 1.31% (GPT-4 Turbo) and 3.79% (Claude 3 Opus) on the full set; Shell-only agent 11.00% on Lite. (high)
- HumanEvalFix pass@1 with GPT-4 Turbo: 87.7% Python, 89.7% JavaScript, 87.9% Java. (high)
- Removing the edit interface drops Lite performance from 18.0% to 10.3%; removing linting to 15.0%; iterative search to 12.0%. (high)
- 51.7% of trajectories contain a failed edit; recovery odds fall from 90.5% to 57.2% after one failure. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 3 (arxiv.org, dated 11 November 2024); the abstract page is arxiv.org. The paper is released under CC BY 4.0.
Relationships
- supports: SWE-bench — the first agent result on the benchmark and the source of the 12.47% figure.
- depends-on: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — the benchmark, task format and RAG baselines it builds on; five authors are shared.
- supports: AI Benchmarks and Evaluation — evidence that scaffold design changes a fixed model's score by several points.
- related: Introducing SWE-bench Verified (OpenAI, 2024) — SWE-Agent is one of the scaffolds OpenAI ran when releasing the Verified subset.
- related: HumanEval — HumanEvalFix, the secondary evaluation, derives from HumanEval.