ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? (Wang et al., 2026)

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? is a benchmark paper posted to arXiv on 11 May 2026 (arXiv:2605.11086, cs.CR). It introduces ExploitGym, a benchmark measuring whether AI agents can turn a vulnerability into a working exploit, and reports an evaluation of seven model–scaffold configurations against it. The benchmark and the paper are the subject of the concept page ExploitGym.

Provenance

• Authors (16): Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, Kurt Thomas, Yan Shoshitaishvili, Wenbo Guo, Jingxuan He, Thorsten Holz, Dawn Song
• Affiliations: UC Berkeley; Max Planck Institute for Security and Privacy; UC Santa Barbara; Arizona State University; Anthropic; OpenAI; Google
• Submitted: 11 May 2026 · DOI: 10.48550/arXiv.2605.11086
• Division of labour, as stated by the authors: "The benchmark design and experimental methodology are developed by the academic authors. Industry partners provided feedback on the benchmark design, facilitated access to their models, and assisted with running select experiments."

The author list is the point on which secondary descriptions of the paper most often diverge. Dawn Song is the last author and the work is led from Berkeley, but it is not a single-laboratory product: three frontier developers are among the affiliations, and two of them ran experiments on their own models.

The task

Each instance supplies a vulnerable codebase with build configuration, a proof-of-vulnerability (PoV) input that triggers a known bug together with a textual description, and an execution environment. The agent must extend the PoV into a working exploit. The success target is unauthorized code execution — "executing code with privileges that should not be obtainable under the intended security model" — chosen as one of the most severe outcomes. By default the upstream patch is withheld from the agent "to simulate realistic exploitation conditions," though each component of the vulnerability information can be included or withheld independently.

Scoring runs in two stages. Each environment contains a dynamically generated flag stored outside the agent's authorized scope and unreachable through any legitimate interface; submitting its value demonstrates unauthorized code execution. Flag capture is described as "a necessary but not sufficient condition for success." An agent-as-a-judge then examines the trajectory to assess whether the agent genuinely used the intended vulnerability rather than succeeding "through an unrelated shortcut, such as exploiting a different, more easily exploitable vulnerability or reproducing a known public exploit."

Composition

The benchmark holds 898 instances across three domains, each with its own security model and mitigation toggles.

Userspace targets are C/C++ projects such as FFmpeg and OpenSSL, rebuilt with sanitizers disabled so that exploitation is possible at all, and served to an unprivileged nobody user with a setuid-root catflag helper guarding the flag. Browser targets are the standalone V8 shell d8 built at the vulnerable revision, patched to remove convenience APIs such as os.system so that flag retrieval requires genuine exploitation; V8 instances are restricted to reports filed after 2024, when the heap sandbox became default, so the sandbox toggle is meaningful. Kernel targets run in per-connection QEMU/KVM virtual machines with the agent's process under an nsjail sandbox and the flag on a raw block device unreachable even to a process holding UID 0 inside a user namespace.

Results

The headline evaluation runs the full benchmark with mitigations disabled under a two-hour wall-clock timeout per task. All agents receive the same user prompt, except Claude Mythos Preview, which had "adjusted prompting through an additional CLAUDE.md file."

These are counts of instances out of 898, not success rates. The Anthropic model results were obtained in collaboration with Anthropic; experiments ran under structured access programs — OpenAI's Trusted Access for Cyber and Anthropic's Cyber Verification Program — with safeguards disabled, which the authors describe as measuring "the capability boundary of frontier models and agents."

Flag capture versus intended-vulnerability success

The gap between flags captured and scored successes is one of the paper's substantive findings rather than a scoring detail.

The two highest-flag models align at 69.5% and 56.7%, meaning 69 and 90 of their flag captures respectively succeeded by a route other than the intended vulnerability. Manual trace inspection identified two recurring patterns: more commonly, the agent found "a nearby but more powerful flaw while analyzing the target vulnerability" and pivoted to it; more rarely, it concluded the provided vulnerability was non-exploitable and searched for entirely new attack surfaces, "sometimes by auditing source code and, in a few instances, by performing dynamic fuzzing." The authors read both as evidence that "frontier agents can independently discover and exploit vulnerabilities in realistic environments, even without prior knowledge of specific flaws."

Effect of mitigations

Re-enabling standard defenses eliminated most successes without eliminating them entirely. Claude Mythos Preview fell from 107 to 25 in userspace, 38 to 17 in V8, and 12 to 3 in kernel; GPT-5.5 fell from 71 to 10, 27 to 3, and 22 to 8. Every other model dropped to zero in every domain except GPT-5.4, which retained 2 userspace and 1 kernel success. Surviving exploits bypassed ASLR "using partial-pointer overwrites and low-bit brute force," escaped the V8 sandbox "via known rendezvous primitives such as Wasm dispatch tables and Irregexp bytecode," and bypassed KASLR "by abusing writable static strings such as modprobe_path and core_pattern, or by relying on side-channel leaks."

Domain difficulty and time scaling

Kernel exploitation separated the frontier configurations most sharply: Claude Mythos Preview and GPT-5.5 achieved 12 and 22 successes, while no other model achieved more than one. The authors attribute the difficulty to the globally shared kernel heap making layout prediction noisy, race-condition timing outside the attacker's control, build-specific behaviour, and constrained debugging, and treat kernel success as "an especially important signal of advanced exploitation capability."

An extended run with a six-hour timeout, restricted on cost grounds to Claude Mythos Preview and Claude Opus 4.6, found contrasting profiles: Opus 4.6 saturated within 30 minutes at roughly 15 successes, while Mythos Preview continued accumulating successes past the two-hour mark without a clear plateau. The authors conclude that the two-hour budget under-counts that model's capability.

The models' success sets diverge: 56 targets were solved only by Claude Mythos Preview and 26 only by GPT-5.5, with 91 shared; the other models contributed 61 successes, of which 4 were unique to them. Including the six-hour runs, the union across all configurations reaches 239 instances.

Safety filters and refusals

With OpenAI's default safety filters re-enabled and default prompting, GPT-5.5 was blocked before making any tool call in 88.2% of cases; in the remainder it averaged 4.4 valid LLM requests but "remain[ed] in the reconnaissance stage and ma[de] no progress towards exploitation." The authors separately recorded 36 refusals from GPT-5.4 and 23 from GLM-5.1 in the main runs, in which the model declined to develop an exploit for safety reasons.

Judge validation

Every successful trajectory was scored by two agent judges — Codex CLI with GPT-5.5 and Claude Code with Claude Opus 4.6. On an expert audit of 58 validation trajectories (30 yes, 28 no), Codex CLI matched the human label 58/58 (Cohen's κ = 1.00) and Claude 56/58 (κ = 0.931), with Claude's errors both in the direction of accepting exploits of unrelated bugs as intended. Across 313 scored tasks excluding the Anthropic-run experiments, the two judges agreed on 294 (93.9%), with disagreements escalated to human reviewers.

Stated limitations

The authors list: no coverage of Windows, iOS, Android or applications running in them; arbitrary code execution as the sole success criterion, which excludes arbitrary read/write primitives, sandbox escape without code execution, and partial progress; failures that may stem from refusal, tool misuse or non-exploitable vulnerabilities rather than difficulty; the absence of ground-truth exploits for every task, which they note also mitigates data-contamination concerns; a single time- and cost-gated attempt per task; and a single instruction set that "may inadvertently favor one model." They also note that no vulnerability-analysis or exploitation-specific tools were provided.

They add that "higher scores on ExploitGym reflect stronger exploitation abilities or risks, but not necessarily better or safer models overall."

Ethics and access

All benchmark data comes from public repositories and vulnerability databases — OSS-Fuzz, OSV, ClusterFuzz, the Chromium Issue Tracker, kernelCTF and syzbot — and "every included vulnerability has been patched upstream prior to inclusion." Where the pipeline surfaces previously unknown vulnerabilities or techniques, the authors state they will follow responsible disclosure and "withhold associated artifacts until patches are available or the standard 90-day disclosure window has elapsed." The paper does not state terms of public release for the benchmark itself.

The concluding position is that "autonomous exploit development by frontier AI agents is no longer a hypothetical capability," that traditional hardening "remains effective but imperfect," and that "today's reliability limitations should not be interpreted as durable safety guarantees."

Relationships

• related: ExploitGym — the concept page tracking the benchmark and its use.
• depends-on: CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (Wang et al., ICLR 2026) — the userspace corpus is sourced through CyberGym; three authors are shared.
• related: Autonomous cyber-agents — the capability the benchmark measures.
• related: Agentic harnesses and capability elicitation — the scaffold-dependence the results show.
• related: Claude Mythos Preview, GPT-5.5 ('Spud'), Claude Opus 4.6 — the evaluated systems.
• related: AI Benchmarks and Evaluation — benchmark design and judging methodology.