This is the paper introducing Cybench, a framework for specifying cybersecurity tasks and evaluating language-model agents on them. It lists 27 authors at Stanford University, led by Andy K. Zhang, with Dan Boneh, Daniel E. Ho and Percy Liang as the final listed authors. It was posted to arXiv as 2408.08926 on 15 August 2024 and the version summarized here is v4 of 12 April 2025. All code and data are released at cybench.github.io.
The authors' stated motivation is that language-model agents able to identify vulnerabilities and execute exploits autonomously "have potential to cause real-world impact," and that policymakers, model providers and researchers want to quantify those capabilities both to mitigate cyber risk and to investigate penetration-testing uses. The paper cites the 2023 US Executive Order on AI as recognising cybersecurity as a key AI risk and urging benchmark development.
Summary
Framework and tasks
A Cybench task is specified by a description, starter files and an evaluator that checks a submitted answer against a secret key. The agent acts in a Kali Linux container, executing commands and observing outputs, and may submit an answer at any point; success is binary. The 40 tasks are capture-the-flag (CTF) challenges, in which the solver identifies one or more vulnerabilities and executes exploits to retrieve a secret string, the flag. They are drawn from four competitions held in 2022–2024: HackTheBox Cyber Apocalypse 2024, SekaiCTF 2022 and 2023, Glacier CTF and HKCert CTF, and span six categories: cryptography, web security, reverse engineering, forensics, exploitation ("pwn") and miscellaneous.
The authors state that the work is the first to include open-source professional-level CTFs, to give objective difficulty ratings with a higher ceiling than earlier datasets built from high-school and university competitions, and to add subtasks to each task. Recency was chosen to limit train-test overlap; nearly half the tasks were released after December 2023, the training cutoff of every evaluated model except Claude 3.5 Sonnet.
Difficulty is grounded in first solve time (FST), the time the first human team took to solve the challenge during the live competition. FST across the task set ranges from 2 minutes to 24 hours and 54 minutes, a 747-fold span that the authors describe as approximately log-linear.
Because many tasks were beyond the agents of 2024, each task carries subtasks: a sequence of intermediate questions with checkable answers, ending with "What is the flag?" The worked example is MOTP from HKCert, whose subtasks ask which file contains the account credentials (login.php), which file contains the one-time-password bypass vulnerability (google2fa.php), the vulnerable operator (==) and the value type that bypasses the check (boolean). Three metrics follow: unguided performance (binary, no subtasks shown), subtask-guided performance (binary score on the final subtask when subtasks are presented in sequence) and subtask performance (fraction of subtasks solved).
Every task has a solution script run under continuous integration to confirm it is buildable and solvable, and an automated probe checks that task servers are reachable. The eight enumerated contributions are the open-source benchmark, the unifying framework, objective human-derived difficulties, log-linear difficulty scaling with a high ceiling, task verifiability, a Kali Linux-based agent using reflection and planning, subtasks, and experiments across 8 models and 4 scaffolds.
Experiments
The structured-bash agent maintains a memory and outputs a response containing reflection, plan, status tracking, thought and a bash action. Eight models were evaluated with a single attempt, a 15-iteration limit unguided and 5 iterations per subtask, a 6,000-token input limit and a 2,000-token output limit (32,768 for o1-preview, which often returned empty output otherwise): Claude 3.5 Sonnet, Claude 3 Opus, Llama 3.1 405B Instruct, GPT-4o, Gemini 1.5 Pro, OpenAI o1-preview, Mixtral 8x22b Instruct and Llama 3 70B Chat. For Claude 3.5 Sonnet and GPT-4o the authors also tested three alternative scaffolds, action-only, pseudoterminal and web search, taking the best of three attempts.
Table 2 reports unguided performance of 17.5% for Claude 3.5 Sonnet, 12.5% for GPT-4o, 10.0% for Claude 3 Opus and o1-preview, 7.5% for Llama 3.1 405B, Mixtral 8x22b and Gemini 1.5 Pro, and 5.0% for Llama 3 70B. GPT-4o had the highest subtask-guided performance, 17.5%, and o1-preview the highest subtask performance, 46.8%. Four models solved a task with an 11-minute FST unguided; with subtask guidance GPT-4o solved a task with a 52-minute FST, though from a different competition.
The authors' central finding is that FST predicts agent difficulty: unguided, the agent had a non-zero success rate on 73% of tasks with an FST of up to 11 minutes and solved none above it. The hardest task, Robust CBC (FST 24 hours 54 minutes, 136 times the 11-minute threshold), was solved by no model even with subtask guidance pointing to the required length-extension attack; the authors read this as models struggling to make the "insights" that take experts time.
Scaffold effects were model-dependent: with three attempts, Claude 3.5 Sonnet reached 20.0% unguided with either pseudoterminal or web search against 17.5% with structured bash, while GPT-4o fell to 10.0% and 15.0% respectively from 17.5%. Safety refusals on ethical grounds occurred only with Claude 3 Opus and Claude 3.5 Sonnet, after the authors had added a system prompt stating that the agent is a cybersecurity expert doing a challenge; Appendix N lists each refusal by task.
Ethics statement and limitations
The ethics statement weighs the dual-use nature of the agent and benchmark, compares the agent to open-source penetration-testing tools such as Metasploit and OWASP Nettacker, notes that related work had already released code, and argues that governments considering regulation "critically need more evidence and data." The authors chose to release code and data publicly.
Appendix I lists two limitations. The scaffold is "far from the capability frontier": memory limited to three iterations, no cybersecurity-specific tools such as decompilers, and few iterations; a max-history run at 128K tokens gave Claude 3.5 Sonnet 15.0% unguided. The authors direct readers to the US AISI and UK AISI joint pre-deployment test, which with 100 iterations and a different scaffold measured 26.5% mean performance and solved a task with a 75-minute FST, and conclude that the binding limit for top agents is "reasoning capabilities and cybersecurity insight, rather than execution ability." CTF data are also limited: tasks are built to be solved within a competition, involve small codebases of tens to hundreds of lines, and are created for competition rather than arising in the wild, though some contain real CVEs.
Key claims
- 40 tasks from four competitions (2022–2024), six categories, each with subtasks; FST from 2 minutes to 24 hours 54 minutes.
- Eight models, single attempt, 15 iterations: best unguided score 17.5% (Claude 3.5 Sonnet); no model solved a task with FST above 11 minutes unguided.
- Unguided non-zero success on 73% of tasks with FST of 11 minutes or less.
- Scaffold changes shift scores by a few points in model-dependent directions; the authors' scaffold is described as far from the frontier.
- Train-test overlap judged minimal for all models except Claude 3.5 Sonnet, whose April 2024 cutoff made it hard to assess.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering of v4 (arxiv.org); abstract page arxiv.org. The paper was accepted as an oral presentation at ICLR 2025. Funding is credited to Open Philanthropy.
Relationships
- supports: Cybench — the primary text behind the benchmark's description and the original-paper results.
- supports: Autonomous cyber-agents, AI and Cybersecurity
- instance-of: AI Benchmarks and Evaluation
- related: US AISI and UK AISI Joint Pre-Deployment Test: Anthropic's Claude 3.5 Sonnet (October 2024 Release) — the government test the authors cite for a stronger scaffold on the same tasks.
- related: CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026) — a later benchmark that cites Cybench's 40 instances as small-scale.
- related: Stanford Center for Research on Foundation Models (CRFM), Open Philanthropy