"Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces" is the paper introducing the Terminal-Bench framework and its Terminal-Bench 2.0 dataset. Mike A. Merrill and Alexander G. Shaw are listed as equal contributors and corresponding authors, followed by Nicholas Carlini and a large author list spanning 44 affiliations, the first of which are Stanford University, the Laude Institute and Anthropic. It was posted to arXiv as 2601.11868 on 17 January 2026. The dataset and harness are published at tbench.ai.
Summary
Motivation and task format
The authors state that current benchmarks "either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models," and choose the terminal because it is text-based, is used for "highly-skilled and valuable work like software engineering, scientific computing, cybersecurity, and machine learning," and has become the standard tool of agents such as Cursor, Codex CLI, Claude Code and Gemini CLI; they cite Anthropic's claim that Claude Code drives $1 billion in run-rate revenue. The benchmark "evaluates whether agents can perform the kind of high-skill work that professionals are paid to do, including configuring legacy systems, reimplementing research papers, and solving general software engineering problems."
A task consists of an instruction, a Docker image, a set of tests, a human-written reference ("oracle") solution and a time limit. Tests check properties of the final container state, not the agent's commands or output, so any approach that reaches an acceptable end state passes. Tasks use the Harbor task format and run in the Harbor harness, which supports Claude Code, Codex CLI, OpenHands and Mini-SWE-Agent as well as Terminus 2, the authors' own single-tool agent that acts only through a headless terminal and serves as a neutral testbed. The same format has been used to adapt 26 other benchmarks. Examples given include parallelising asynchronous jobs with correct cleanup on keyboard interrupt, rewriting a COBOL program in Python with identical input-output behaviour, building Linux from source, reverse-engineering binaries, and fixing the OCaml garbage collector after a failed optimisation.
Construction and verification
Tasks were crowd-sourced: 93 contributors wrote 229 tasks, from which 89 were selected for Terminal-Bench 2.0 on the basis of the author's difficulty assessment and review by three experienced reviewers. Verification targets three criteria: specificity (tests pass if and only if the container ends in an acceptable state), solvability (the oracle solution passes) and integrity (no shortcut unavailable in real deployment, for instance future git commits stripped from a repository). The process comprised an automated run of the oracle solution and checks that a no-op agent fails, a contributor checklist, a language-model check tool, manual review with the contributor, runs with several models to distinguish capability failures from task defects, an adversarial exploit agent hunting for ways to pass without solving, and a final review by two further auditors. The authors estimate about three reviewer-hours per task, "multiple hundreds of person-hours" in total, and acknowledge the benchmark "may still have flaws."
Contributors estimated completion times: experts would need under an hour for 36 tasks (48.6%), one hour to one day for 35 (47.3%) and one day to one week for 3 (4.1%); juniors under an hour for 6 (8.1%), one hour to one day for 53 (71.6%), one day to one week for 12 (16.2%) and over a week for 3 (4.1%). The fix-ocaml-gc task is estimated at almost a day for an expert and ten days for a junior. Software engineering is the largest author-assigned category but no category holds a majority.
Results
Six agents (Claude Code, Codex CLI, Gemini CLI, OpenHands, Mini-SWE-Agent, Terminus 2) and 16 models were run at least five times per combination, 32,155 trials in all, with configurable reasoning effort left at provider defaults ("medium" for Anthropic and OpenAI), first-party APIs for closed models and Together.AI for open-weight models. The best resolution rate was Codex CLI with GPT-5.2 at 62.9% (±3.0), followed by Terminus 2 with Claude Opus 4.5 at 57.8% (±2.5) and with Gemini 3 Pro at 56.9% (±2.5); Terminus 2 with GPT-5.2 scored 54.0%, Claude Code with Opus 4.5 52.1%, Gemini 3 Flash 51.7%, Codex CLI with GPT-5 49.6%, Terminus 2 with Claude Sonnet 4.5 42.8%, Claude Opus 4.1 38.0%, Kimi K2 Thinking 35.7% (the best open-weight result), Gemini 2.5 Pro 32.6%, Claude Haiku 4.5 29.8% with Mini-SWE-Agent, Kimi K2 Instruct 27.8%, GLM 4.6 24.5%, Qwen 3 Coder 480B 24.3% with OpenHands, GPT-OSS-120B 18.7%, GPT-5-Nano 11.5% with Codex CLI, and GPT-OSS-20B 3.4%. Proprietary models occupied the top 13 positions. Switching Codex CLI from GPT-5-Nano to GPT-5.2 raised its rate by 52 points, while moving Gemini 2.5 Pro from OpenHands to Terminus 2 raised it by 17, which the authors read as model choice mattering more than scaffold. Some tasks were solved by no model.
A full run costs between one and a hundred dollars depending on the model. Most attempts finish within 20 minutes, but some last two hours with hundreds of API calls and almost 100 million tokens; neither turn count nor token count correlates with success. State-of-the-art performance nearly doubled in the eight months between Gemini 2.5 Pro and GPT-5.2, and the authors expect the dataset "may become saturated within the next year." Human difficulty labels correlate with an "empirical difficulty" derived from Terminus 2 pass rates (r = 0.436, p < 0.001): 93.3% of human-rated hard tasks were empirically hard, while 54.5% of human-rated medium tasks were empirically hard, often those needing adversarial reasoning such as XSS filter bypasses or Core War strategy.
Failure analysis used a taxonomy adapted from MAST with three classes, execution, coherence and verification, judged by GPT-5 at 90% agreement with 120 human-labelled traces; execution errors dominated for Opus 4.5 and GPT-5.2, while Qwen Coder showed a more balanced profile. At the command level, error rates ranged from 9.2% (Grok 4) to 26.7% (GPT-OSS-120B), with calls to executables not installed or not in PATH the most common failure (24.1%).
Limitations
Agents have internet access, so in principle one could find the dataset and read oracle solutions; the authors say they have not observed this in tens of thousands of trajectories. The Big-Bench canary string is included in every repository file, but the authors note that options against deliberate training on the data are limited and leave a private test set to future work. Internet access and machine variability also limit reproducibility.
Key claims
- Terminal-Bench 2.0: 89 tasks selected from 229 contributed by 93 people, each verified by three reviewers at roughly three reviewer-hours per task. (high)
- Best results: Codex CLI + GPT-5.2 62.9%; Terminus 2 + Claude Opus 4.5 57.8%; Terminus 2 + Gemini 3 Pro 56.9%; best open-weight, Terminus 2 + Kimi K2 Thinking, 35.7%. (high)
- 32,155 trials across six agents and 16 models; proprietary models hold the top 13 places. (high)
- Expert time estimates: 48.6% under an hour, 47.3% one hour to one day, 4.1% one day to one week. (high)
- Command-level error rates 9.2–26.7%; missing executables are 24.1% of command failures. (high)
- The authors expect saturation within a year and note the Big-Bench canary string as the only contamination control. (high)
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of version 1 (arxiv.org, dated 17 January 2026); the abstract page is arxiv.org. Experiment configurations are at github.com/laude-institute/terminal-bench-experiments.
Relationships
- supports: Terminal-Bench — the primary text for the task format, verification process and launch results.
- depends-on: SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (Yang, Jimenez et al., 2024) — Mini-SWE-Agent, one of the evaluated scaffolds, descends from SWE-agent.
- related: Detecting Safety Violations Across Many Agent Traces (Stein, Brown et al., University of Pennsylvania, 2026) — later audit that found scaffold-level cheating in top leaderboard submissions.
- related: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — the repository-level benchmark the authors cite as precedent.
- supports: AI Benchmarks and Evaluation — outcome-based verification and the adversarial exploit-agent check as design practices.