This is a peer-reviewed conference paper by Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang and Dawn Song of the University of California, Berkeley, introducing the CyberGym benchmark. Wang and Shi are marked as equal contributors. It was first posted to arXiv as 2506.02548 on 3 June 2025, revised twice (v2 on 8 October 2025, v3 on 24 March 2026), and published as a conference paper at the Fourteenth International Conference on Learning Representations in 2026.
The paper's stated purpose is to correct two limitations the authors identify in existing cybersecurity benchmarks for AI agents: that they are small-scale — "up to 200 instances" — because they depend on heavy manual construction or brittle data sources, and that they measure only static, historical instances, which the authors argue "mak[es] it difficult to determine how AI agents impact constantly evolving, current cybersecurity landscape." CyberGym addresses the first with scale and the second by running its agents against current software to find previously unknown flaws.
A footnote in the paper's introduction records that CyberGym "has been adopted in the system cards of various frontier models for cybersecurity evaluation, such as Claude, Kimi, and GLM," which is the paper's own account of the cross-lab role the benchmark came to play.
Benchmark construction
CyberGym contains 1,507 instances derived from real vulnerabilities across 188 widely used open-source projects. The vulnerabilities are sourced from OSS-Fuzz, Google's continuous fuzzing service, which the paper describes as having discovered over 13,000 vulnerabilities across more than 1,000 open-source projects since its 2016 launch. Of the 1,507 instances, 1,368 come from the ARVO dataset of OSS-Fuzz findings packaged as reusable Docker images (covering vulnerabilities up to 31 July 2024) after passing the authors' quality-assurance pipeline; a further 139 more recent vulnerabilities were collected directly. The full set covers vulnerabilities disclosed between 1 January 2017 and 21 April 2025.
The authors scope the benchmark to memory-safety vulnerabilities in C/C++ projects detectable through compiler sanitizers, on the grounds that memory-safety flaws represent "more than 70% of high-severity vulnerabilities in industry reports from Google, Microsoft, and Mozilla" and that the domain supplies reliable tooling — sanitizers as a detection oracle and OSS-Fuzz as a corpus. The paper states the limitation this creates directly: the memory-safety focus "limits us to primarily C/C++ codebases, which may not capture the full landscape of security vulnerabilities."
Quality control involved re-running each ground-truth proof-of-concept on pre-patch and post-patch executables to confirm reproducibility, and excluding instances where multiple entries shared a patch commit or where executables had similar logic, identified by comparing crash stack traces.
The benchmark's reported statistics show wide variation across instances. Vulnerability descriptions have a median length of 24 words, reaching up to 158. Ground-truth proof-of-concept inputs range from a few bytes to over 1 MB. Codebases have a median of 1,117 files and 387,491 lines of code, spanning from tens of thousands to millions of lines. Patches modify a median of 1 file and 7 lines, but reach up to 40 files and 3,456 lines in the most complex cases. The projects span networking (cURL), cryptography (OpenSSL), programming tools (GNU Binutils), scientific computing (OSGeo), operating systems (QEMU) and multimedia (FFmpeg); the most starred, OpenCV, has over 80,000 GitHub stars. The distribution across projects is long-tailed, with 62.4% of instances drawn from projects outside the top 10, and covers 28 distinct sanitizer crash types.
Table 1 of the paper positions CyberGym against prior work: NYU CTF Bench (200 instances), Cybench (40), AutoAdvExBench (75 instances, 41 projects), CVE-Bench (40 instances, 26 projects), BountyBench (40 instances, 31 projects) and SEC-bench (200 instances, 29 projects). CyberGym is described as "over seven times larger than any other cybersecurity benchmark," and is the only entry in the table marked as producing zero-day findings.
Task formulation
In the primary task, the agent receives a text description of a historically found vulnerability — including approximate location, type and root cause — together with the codebase as it stood before the vulnerability was patched, and a pre-patch executable in a containerized environment. The agent must produce a proof-of-concept input that reproduces the vulnerability, submitting candidates via a bash script and iterating on execution feedback such as exit code and command-line output.
The authors chose reproduction as the main task both because success can be verified by execution rather than judged, and because it is difficult in practice: they cite findings that human security experts require approximately 5 hours to reproduce known vulnerabilities from public reports, and that automated fuzzing tools take a median of 324 days to reveal vulnerabilities in real-world OSS-Fuzz projects.
Four difficulty levels vary the information supplied. Level 0 withholds the text description entirely, simulating open-ended discovery; level 1, the primary task, supplies the description; level 2 adds a stack trace; level 3 adds the ground-truth patch.
Evaluation results
The paper evaluates four agent frameworks and eleven frontier language models, at a stated cost of over $40,000 in API credits and 1,000 H100 GPU hours. Running state-of-the-art agents in non-thinking mode across the full benchmark costs approximately $3,000 in API credits, which prompted the authors to publish a randomly selected 300-instance subset for lighter-weight evaluation.
With OpenHands as the scaffold and thinking mode disabled, level-1 success rates were: Claude-Sonnet-4 17.9%, Claude-3.7-Sonnet 11.9%, GPT-4.1 9.4%, GPT-5 (minimal reasoning) 7.8%, Gemini-2.5-Flash 4.8%, DeepSeek-V3 3.6%, o4-mini 2.5%, R2E-Gym-32B 2.0%, Qwen3-235B-A22B 1.9%, OpenHands-LM-32B 1.7% and SWE-Gym-32B 0.1%. The union of all model results reaches 27.2%, which the authors read as evidence of low overlap in the tasks different models solve.
Enabling thinking mode produced modest gains for most models but moved GPT-5 from 7.7% to 22.0%, surpassing Claude-Sonnet-4. The paper attributes o4-mini's low score to a behavioral pattern rather than capability: the model "often conservatively requests user confirmation and prematurely terminates the execution," which the authors do not observe in the other models and present as a case where agent developers must handle model-specific behavior.
Models fine-tuned for SWE-bench — SWE-Gym-32B, R2E-Gym-32B and OpenHands-LM-32B — scored at or below 2.0%, which the paper presents as evidence that CyberGym is complementary to rather than redundant with software-engineering benchmarks. The authors attribute the gap to CyberGym requiring repository-wide reasoning to trace a path from a program's entry point to the vulnerability, where SWE-bench tasks often involve localized code changes.
Comparing four agent frameworks on a common GPT-4.1 backbone at roughly $2.00 per task, success rates clustered closely — OpenHands 9.4%, Cybench agent 9.0%, Codex 7.4%, EnIGMA 7.2% — but the union across all four reached 18.4%, nearly double the best individual result.
Success rates fall sharply with the length of the ground-truth proof-of-concept. Instances requiring inputs under 10 bytes were solved at 55.3% by Claude-Sonnet-4 and 43.5% by GPT-4.1, while instances requiring inputs longer than 100 bytes — 65.7% of the benchmark — were solved at around 10%. Difficulty-level ablation with OpenHands and GPT-4.1 gave 3.5% at level 0, 9.4% at level 1, 13.1% at level 2 and 17.1% at level 3.
The authors tested for training-data contamination by splitting instances on each model's knowledge cutoff and comparing success rates with Fisher's exact test and a two-proportion Z-test. All p-values exceeded 0.1, indicating no statistically significant difference: Claude-3.7-Sonnet 11.9% pre-cutoff against 12.5% post-cutoff, GPT-4.1 9.7% against 5.6%, GPT-5 (minimal) 7.7% against 8.0%, o4-mini 2.4% against 2.8%.
Recurring failure patterns identified include exhausting iteration limits on unproductive attempts, prematurely requesting user information instead of inferring from code, overwhelming the context window with excessive output, premature or incorrect success declarations, and repeated retrieval failures.
Zero-day discovery
The paper's second contribution is that the benchmark produced findings against current software. Among all proof-of-concept inputs generated during the evaluation, 759 triggered sanitizer crashes on post-patch versions across 60 projects — meaning they triggered some flaw other than the one the agent was asked to reproduce. Validating those 759 against the latest versions of their programs left 35 that still crashed; manual root-cause analysis and deduplication reduced these to 9 unique previously unreported vulnerabilities. The authors measured how long these had existed by the interval between the earliest version in which presence was confirmed and the latest version, giving an average of 969 days.
Separately, fuzzy matching of sanitizer reports from ground-truth pre-patch runs against generated post-patch runs, followed by manual inspection of close matches, identified 18 incomplete patches across 15 projects — cases where the original fix did not fully address the vulnerability. One affects the latest version of its project. The authors updated the benchmark's post-patch versions in these cases to the first version where the vulnerability is fully addressed.
For open-ended discovery, the authors ran OpenHands with GPT-4.1 and GPT-5 at difficulty level 0 against the latest versions of 431 OSS-Fuzz-supported projects containing 1,748 entry executables. GPT-4.1 triggered 16 crashes yielding 7 confirmed unique zero-days; GPT-5, at high reasoning effort, triggered 56 crashes yielding 22, with 4 overlapping between the two models — 25 unique in total.
All vulnerabilities were disclosed to project maintainers, with the authors stating they would wait for patches or a 90-day responsible-disclosure period before public release. As of writing, 4 CVE assignments had been received and 10 vulnerabilities patched.
The 34-versus-35 discrepancy
The paper reports its zero-day total inconsistently, and the inconsistency is internal to the ICLR 2026 camera-ready rather than a difference between publication venues. The abstract and Section 5 both state 34 zero-day vulnerabilities and 18 incomplete patches, and the arithmetic of Section 5 supports that figure: 9 unique zero-days from the benchmarking runs plus 25 from open-ended discovery. The paper's own introduction, however, states that analysis of the generated proof-of-concept inputs "reveal 17 inadequate historical patches and 10 previously unknown vulnerabilities," which together with the 25 from open-ended discovery gives 35 zero-days and 17 patches.
The Berkeley project page's figures of 35 zero-days and 17 incomplete patches (Source: rdi.berkeley.edu) match the introduction's numbers rather than the abstract's. The published version's abstract, Section 5 and the contributions list all carry 34 and 18, so those are the figures the camera-ready asserts; the introduction appears not to have been updated alongside the rest. Neither the paper nor the project page notes the divergence.
Limitations stated by the authors
Beyond the memory-safety and C/C++ scoping limitation noted above, the authors caution that reported scores depend on the agent scaffold as much as on the model: the union-of-agents result of 18.4% against a best-individual 9.4% on a fixed backbone shows that scaffold choice materially changes measured capability. The paper does not claim its evaluated configurations are optimal, and later frontier-model reporting of much higher CyberGym scores is not directly comparable to the figures here, which were produced with 2025-era models and harnesses.
Provenance
Retrieved 15 August 2026 from the arXiv PDF for version 3 (arxiv.org), the ICLR 2026 camera-ready. The abstract page (arxiv.org) records the submission history and the DOI 10.48550/arXiv.2506.02548; the OpenReview identifier is 2YvbLQEdYt. Two citation years appear in the literature — wang2025cybergym for the arXiv preprint and wang2026cybergym for the ICLR paper — referring to the same work. The benchmark's own site is cybergym.io.
Relationships
- supports: CyberGym — the primary text behind the benchmark's description.
- supports: Autonomous cyber-agents — supplies the measurements the capability claims rest on.
- instance-of: AI Benchmarks and Evaluation — an execution-scored agentic benchmark.
- related: AI and Cybersecurity — the policy area the findings bear on.
- related: Pacing the Frontier (statement from employees of frontier AI companies, July 2026) — Dawn Song's statement on the benchmark's policy significance.
- related: Claude Mythos Preview System Card, GPT-5.6 Preview System Card (OpenAI, June 2026), Muse Spark Safety & Preparedness Report (Meta, May 2026) — frontier reports that adopt the benchmark.