CyberGym

CyberGym is a benchmark for the vulnerability-discovery capabilities of AI agents, built at the University of California, Berkeley. It consists of 1,507 benchmark instances derived from real vulnerabilities in 188 widely used open-source projects, and asks an agent working from the project's source code to identify a flaw and produce an input that demonstrably triggers it (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026); Source: rdi.berkeley.edu).

The benchmark occupies a specific position in the evaluation stack that developed around autonomous cyber agents during 2026. CyberGym measures the finding end of the offensive-security chain — locating and validating a flaw — while companion benchmarks from the same group measure the exploitation end, where a validated flaw must be converted into a working attack. That division is why 2026 model comparisons frequently show a model at or near the top of CyberGym while trailing substantially on exploitation measures.

Construction

The benchmark instances were assembled from vulnerabilities surfaced by OSS-Fuzz, Google's continuous fuzzing campaign against open-source software, which the authors selected in order to keep the tasks grounded in flaws that occurred in deployed code rather than in synthetic test cases (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)). Of the 1,507 instances, 1,368 derive from the ARVO packaging of OSS-Fuzz findings and 139 were collected directly to improve timeliness; the vulnerabilities were disclosed between 1 January 2017 and 21 April 2025. Each instance supplies the agent with the project source in a white-box configuration together with a pre-patch executable in a container; success is scored on whether the agent produces an input that reproduces the fault, rather than on a written description of the vulnerability. Four difficulty levels vary the information given, from level 0 (codebase only, simulating open-ended discovery) through level 1 (the primary task, with a text description) to levels 2 and 3, which add a stack trace and the ground-truth patch (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)).

The benchmark is scoped to memory-safety vulnerabilities in C/C++ projects detectable by compiler sanitizers. The authors justify this by the prevalence of memory-safety flaws — more than 70% of high-severity vulnerabilities in industry reports from Google, Microsoft and Mozilla — and by the availability of reliable tooling, while stating the resulting limitation that the scope "may not capture the full landscape of security vulnerabilities" (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)).

Beyond scoring against known vulnerabilities, the authors ran the benchmark's agents in an open-ended discovery configuration against the latest versions of 431 OSS-Fuzz-supported projects. The published paper reports 34 previously unknown vulnerabilities and 18 historically incomplete patches, comprising 9 unique zero-days surfaced incidentally during the benchmarking runs and 25 from open-ended discovery, with 4 CVE assignments received and 10 vulnerabilities patched as of writing (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)).

The 35-zero-days-and-17-patches figure carried by the Berkeley project page (Source: rdi.berkeley.edu) traces to a discrepancy internal to the paper: its introduction states 17 inadequate patches and 10 zero-days from the benchmarking runs, which with the 25 from open-ended discovery gives 35 and 17, while the abstract, Section 5 and the contributions list all give 34 and 18. The published version's arithmetic supports 34 and 18, and the introduction appears not to have been revised in step (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)).

The benchmark was introduced in "CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale" by Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang and Dawn Song, posted to arXiv as 2506.02548 in June 2025 and published at the Fourteenth International Conference on Learning Representations in 2026 (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)). Song's group at Berkeley's Center for Responsible Decentralized Intelligence maintains a public leaderboard and a broader Frontier AI Cybersecurity Observatory tracking attack- and defense-side scores (Source: rdi.berkeley.edu).

At publication the benchmark was hard for the models then available. With OpenHands as the scaffold and thinking disabled, the best backbone was Claude-Sonnet-4 at 17.9%; enabling high reasoning effort moved GPT-5 from 7.7% to 22.0%. Models fine-tuned for SWE-bench scored at or below 2.0%, which the authors present as evidence that CyberGym measures something the software-engineering benchmarks do not (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)). The 2026 frontier scores in the table below, all above 40%, were produced with later models and different harnesses and are not comparable to these figures.

Companion benchmarks

Two related evaluations from the same research programme appear alongside CyberGym in 2026 model reporting and measure later stages of the same chain. ExploitBench requires root-cause reasoning about a real vulnerability and the construction of a working exploit. ExploitGym, which the Pacing the Frontier statement identifies Song as having developed at Berkeley RDI, counts how many exploitation tasks a model completes within time-normalized budgets. Song's comment accompanying that statement frames the pair together: "CyberGym and ExploitGym show that frontier AI agents are now capable of discovering and exploiting real-world software vulnerabilities, which without appropriate safeguards, could enable cyberattacks at scale" (Pacing the Frontier (statement from employees of frontier AI companies, July 2026)).

Use in frontier evaluations

CyberGym is reported in the system cards and safety reports of four developers, which makes it one of the few cyber evaluations with cross-lab comparability, though the scores are produced in each developer's own harness and are not run by a common evaluator.

Anthropic reports CyberGym in the Claude system-card series. The Opus 4.6 card lists it among the domain evaluations; the Opus 4.7 card names it alongside Cybench and Firefox exploit tasks in the Frontier Red Team's assessment; and the Opus 4.8 card reports it alongside ExploitBench and OSS-Fuzz while recording that Cybench was dropped on the basis of earlier findings. The Mythos system card reports 83.1% against 66.6% for Opus 4.6, a 16.5-percentage-point gap, and states that the model "surpasses all but the most skilled humans at finding and exploiting software vulnerabilities" — the finding that triggered the High Cybersecurity threshold under Anthropic's responsible scaling policy.

OpenAI uses the benchmark to measure jailbreak transfer rather than headline capability alone. The GPT-5.6 preview system card reports that one universal jailbreak discovered during automated red-teaming preserved most of the model's cyber task capability when run without blocking safeguards, at 83.0% task success against 83.6% without the jailbreak, succeeded on 10.0% of tasks against the full safeguard stack during the initial internal campaign, and fell to 0% after further mitigations.

Meta reports CyberGym in support of a capability-gap argument. The Muse Spark safety and preparedness report rates cybersecurity risk at moderate or lower on the basis that the model scores 43.5% against 66.6% for Opus 4.6, and reproduces a model trace articulating the conflict the task creates: "my safety alignment prefers to avoid generating functional exploit inputs, while the evaluation incentivizes maximizing success at crashing the vulnerable program – rewarding precisely the harmful-capability behavior I am trained to limit."

Z.ai reports the benchmark for the GLM line and against Western frontier models. The company's own note on the GLM-5.1 results records that some competing models occasionally refused tasks they flagged as security risks, which Z.ai said may lower those models' scores (Source: z.ai).

Reported scores

Figures are self-reported by the developer named unless otherwise stated, and are not run in a common harness; entries from different reporting events are not directly comparable.

One label discrepancy runs through the August 2026 Z.ai reporting. The prose of Z.ai's GLM-5.3 announcement attributes the 83.8% figure to Mythos 5, and Reuters reported it the same way, while the accompanying benchmark table in the same announcement labels that column "Fable 5 (w/ fallback)" (Source: z.ai; Source: reuters.com). Anthropic has not published a CyberGym figure for Mythos 5 confirming either reading.

Limitations noted in reporting

Three limitations recur across the sources that report CyberGym results, and the first is documented in the paper itself: running four agent frameworks on a fixed GPT-4.1 backbone produced closely clustered individual success rates of 7.2–9.4%, but a union across the four of 18.4%, so scaffold choice changes measured capability by roughly a factor of two (CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026)). Harness variation is correspondingly the most consistent limitation in developer reporting: developers run the benchmark under different agent scaffolds, reasoning-effort settings, timeout policies and network allowlists, and Z.ai's August 2026 footnotes disclose evaluating GLM-5.3 inside Claude Code 2.1.207 at maximum reasoning effort with unlimited per-task timeouts and single-run pass@1 scoring across all 1,507 tasks (Source: z.ai). Refusal behavior is the second: a safety-trained model that declines a task scores as a failure, which Z.ai has said may depress competing models' figures and which the Meta trace illustrates from the model's side. Independent reproduction is the third: where a model's weights are not public and no third-party evaluator has run the benchmark, the reported score rests on the developer's own run, a gap Z.ai's own coverage acknowledged for GLM-5.3 pending the open-weight release (Source: marktechpost.com).

Relationships

• depends-on: CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026) — the paper introducing the benchmark.
• instance-of: AI Benchmarks and Evaluation — the general evaluation frame.
• supports: Autonomous cyber-agents — supplies the capability measurements the concept rests on.
• related: AI and Cybersecurity — the broader policy area.
• related: Open-Weight Frontier Models — the benchmark is the measure at issue in open-weight cyber-capability comparisons.
• related: Shadow Evaluations, Unverbalized Evaluation Awareness — evaluation-integrity concerns visible in reported CyberGym traces.
• related: Claude Mythos 5, GPT-5.6 (Sol, Terra, Luna), GLM-5.3, Muse Spark (Meta Superintelligence Labs) — models whose cyber assessments turn on this score.
• depends-on: Pacing the Frontier (statement from employees of frontier AI companies, July 2026) — Song's framing of the benchmark's policy significance.