This paper introduces AgentHarm, a benchmark measuring whether a language-model agent can be directed to complete malicious multi-step tasks. Its authors are Maksym Andriushchenko, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter and Matt Fredrikson of Gray Swan AI, and Alexandra Souly, Eric Winsor, Jerome Wynne, Yarin Gal and Xander Davies of the UK AI Security Institute, the latter five marked as core contributors; additional affiliations listed are EPFL, the Center for AI Safety, Carnegie Mellon and Oxford. It was posted to arXiv as 2410.09024 on 11 October 2024, last revised 18 April 2025, and appeared at ICLR 2025. The dataset is published on Hugging Face by the institute.
The paper's premise is that jailbreak robustness has been studied almost entirely for models acting as chatbots, while agents that call external tools and execute multi-stage tasks may pose greater misuse risk, and that single-turn robustness does not necessarily transfer to the multi-step setting.
Summary
Dataset
AgentHarm contains 10 base behaviours in each of 11 harm categories — fraud, cybercrime, self-harm, harassment, sexual content, copyright, drugs, disinformation, hate, violence and terrorism — giving 110 base behaviours and 440 after augmentation, split 44 validation, 264 public test and 132 private test. It uses 104 unique functions, averaging 3.53 per behaviour, and a public-test run consumes roughly 0.5 to 1.5 million tokens.
Tasks are written to be explicitly malicious rather than dual-use, to require several function calls with ordering dependencies so that later calls use information from earlier ones, and to be digitally realisable. On ethical grounds the authors exclude real names, addresses and places. Each base behaviour appears in four forms: the behaviour presented directly ("detailed"), an open-ended variant omitting details, and hint-included versions of each naming the function to call first.
A parallel benign dataset rewrites each harmful behaviour on the same topic without the malicious intent, so that a model's tool-use competence can be measured without refusals depressing the score.
All tools are synthetic, implemented as side-effect-free standalone functions in the institute's Inspect framework, ranging from general-purpose (search, email, terminal) to domain-specific. Most support self-correction, returning an error message the agent can act on. The authors state plainly that synthetic tools make the tasks easier and less realistic, "which means they act as proxies for harm as opposed to directly indicating harmful agent abilities," while keeping the dataset safe and cheap to run.
Grading is mostly predefined rule-based checks against the agent's calls and arguments, with partial credit, and a semantic model judge used only where a portion of an output must be judged for meaning. The authors argue that grading a whole output with a model is problematic because of judge bias, poor calibration and attacks on the judge, and that restricting the judge to narrow checks reduces this. A separate refusal judge is applied to every message the agent produces, since a refusal can arrive after several tools have run. The authors note that open-ended criteria mean a fully successful run can still score below 100%.
Design principles
The authors state five. Coverage across harm categories was preferred over depth in one domain, on the grounds that performance and refusals vary by domain and that existing domain-specific agentic benchmarks hold roughly 10 to 30 tasks against AgentHarm's 110 base and 440 augmented. Capability degradation is detected by scoring task success rather than non-refusal alone, citing the StrongREJECT finding that jailbreaks can leave models incoherent. Usability is served by shipping in Inspect with isolated tools. Scoring reliability comes from writing every question by hand after finding model-generated behaviours low-quality and unrealistic, reviewing each with a second human reviewer, and manually inspecting many samples. Contamination is addressed with a canary string and with a private set never exposed to public APIs, evaluated on the authors' own Azure deployment.
Results
Evaluation used direct prompting in a simple loop with no elaborate scaffolding, temperature 0, up to 4,096 output tokens, and GPT-4o as the semantic and refusal judge. Models covered were GPT-3.5 Turbo, GPT-4o mini and GPT-4o; Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus and Claude 3.5 Sonnet; Gemini 1.0 Pro, 1.5 Flash and 1.5 Pro; Mistral Small 2 and Large 2; and Llama 3.1 at 8B, 70B and 405B. About 16% of behaviours were skipped for Gemini models because of an API error on argument-free function calls, and the Llama models needed a special prompt to produce syntactically valid tool calls.
Two attacks were tested alongside direct requests: forced tool calls, using providers' own forced-tool-calling option, which was unavailable for the Gemini and Llama deployments; and a single universal rule-based jailbreak template adapted to the agentic setting, tuned on one private-validation sample and applied identically across models in a single attempt.
Without any attack, several models complied widely. Mistral Large 2 refused 1.1% of requests and scored 82.2%, the highest unattacked score; GPT-4o mini scored 62.5% while refusing 22%. Claude 3.5 Sonnet scored 13.5% with 85.2% refusals, Gemini 1.5 Pro 15.7% with 78.5%, and Llama 3.1 405B refused most of all at 88.3%.
The template attack raised scores substantially: GPT-4o from 48.4% to 72.7% with refusals falling from 48.9% to 13.6%; Claude 3.5 Sonnet from 13.5% to 68.7% with refusals from 85.2% to 16.7%; Gemini 1.5 Pro from 15.7% to 56.1% with refusals from 78.4% to 3.5%. Comparing harm scores on non-refused behaviours with the benign set shows little capability loss: under the template attack Claude 3.5 Sonnet scored 81.9% on non-refused behaviours against 82% on benign tasks, and GPT-4o 84.2% against 89.9%.
Limitations stated by the authors
Prompts are English only; only the first user message is adversarial, with no multi-turn follow-up; grading functions can miss valid alternative execution traces; the benchmark requires its own custom tools, since grading is tied to them, so third-party tools and scaffolds are hard to substitute; and the tasks measure basic agentic competence rather than the advanced autonomous capabilities that open-ended agent evaluations target.
Key claims
- 110 base behaviours, 440 augmented, 11 harm categories; 44 validation / 264 public test / 132 private test.
- Scoring combines rule-based criteria with a narrow semantic judge, plus a separate refusal judge over all agent messages; a parallel benign set isolates capability from refusal.
- Unattacked: Mistral Large 2 82.2% harm score at 1.1% refusals; Claude 3.5 Sonnet 13.5% at 85.2% refusals; Llama 3.1 405B refused 88.3%.
- One universal template: GPT-4o 48.4% to 72.7%; Claude 3.5 Sonnet 13.5% to 68.7%; Gemini 1.5 Pro 15.7% to 56.1%.
- Jailbroken agents retain tool-use competence, unlike the capability degradation reported for chatbot jailbreaks.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org, first posted 11 October 2024, revised 18 April 2025. Dataset at huggingface.co.
Relationships
- supports: AgentHarm — the primary text behind the benchmark's description and its original results.
- depends-on: A StrongREJECT for Empty Jailbreaks (Souly et al., 2024) — the capability-degradation finding the benchmark's scoring design responds to.
- instance-of: AI Benchmarks and Evaluation
- related: UK AI Safety Institute (AI Security Institute), Center for AI Safety (CAIS)
- related: HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (Mazeika et al., 2024) — a chatbot-setting red-teaming framework the authors distinguish their agentic tasks from.