This paper introduces HarmBench, an evaluation framework for automated red teaming of language models. Its corresponding author is Mantas Mazeika (University of Illinois Urbana-Champaign), with Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Nathaniel Li, Steven Basart and Dan Hendrycks of the Center for AI Safety, Norman Mu (UC Berkeley), Elham Sakhaee (Microsoft), and Bo Li and David Forsyth (UIUC). It was posted to arXiv as 2402.04249 on 6 February 2024 and released at github.com/centerforaisafety/HarmBench.
The paper's premise is that automated red teaming papers report results under incompatible setups, making them hard to compare, and that prior evaluations lack properties needed to measure red teaming accurately. The authors name three: breadth of behaviours, comparability across papers, and metrics robust to the optimisation pressure that codeveloping attacks and defences creates.
Summary
Problem framing
Red teaming is formulated as designing test cases to elicit a specified behaviour from a target model. The primary measure is attack success rate (ASR), the share of test cases that elicit the behaviour, computed with greedy decoding and a classifier mapping a completion and a behaviour to success or failure.
The authors identify a parameter that prior work left unstandardised: the number of tokens the target model generates during evaluation, which they show can change substring-matching ASR by up to 30 percentage points, so that "cross-paper comparisons can be misleading." They standardise it at 512 tokens and use a metric less sensitive to it. They also propose a prequalification test for classifiers on three non-standard cases — a completion that refuses then proceeds, a random benign paragraph, and a completion for an unrelated harmful behaviour — and report that several previously used classifiers fail it. They note that some prior work evaluates on the same metric its method optimises, which invites gaming.
The benchmark
HarmBench contains 510 unique behaviours, 400 textual and 110 multimodal, with an official split of 100 validation and 410 test behaviours; attacks and defences are required not to tune on the test set or semantically identical behaviours.
Behaviours carry two categorisations. The seven semantic categories are cybercrime and unauthorised intrusion, chemical and biological weapons or drugs, copyright violations, misinformation and disinformation, harassment and bullying, illegal activities, and general harm. The four functional categories are standard behaviours (200), modelled on AdvBench and the TDC 2023 red-teaming dataset; copyright behaviours (100), scored by a hashing classifier that checks whether copyrighted text was actually produced; contextual behaviours (100), which pair a context string with a behaviour referring to it; and multimodal behaviours (110), which pair an image with a behaviour referring to it.
Curation began by distilling the acceptable-use policies of OpenAI, Anthropic, Meta and Inflection AI into a combined summary, from which authors wrote candidate behaviours. Two considerations shaped the set. The first is differential harm: whether a behaviour is materially harder to accomplish with a search engine than with a model. In a small check, one author spent ten minutes searching 20 randomly sampled behaviours from each of three datasets and found 55% searchable for MaliciousInstruct, 50% for AdvBench and 0% for HarmBench contextual behaviours. The second is dual intent: behaviours a developer would plausibly want a model to perform for legitimate reasons were filtered out, since a benchmark that penalises them measures the wrong thing.
The pipeline runs in three steps: the red-teaming method generates test cases for each behaviour, the target model generates completions, and a classifier scores them. A test case counts as successful if the completion exhibits the behaviour or clearly attempts to, which the authors adopt to separate a model's safety measures from its competence — an attempt containing incorrect code still counts. The non-copyright classifier is a fine-tuned Llama 2 13B chat model, which the authors report outperforms existing classifiers on their labelled validation set and is the only open-source one reaching acceptable performance; they argue against closed-source classifiers because they can change without notice or be withdrawn.
Results
Eighteen red-teaming methods from twelve papers were run against 33 models, 24 open-source and 9 closed-source. The text methods include GCG and two variants, PEZ, GBDA, UAT, AutoPrompt, Stochastic Few-Shot, Zero-Shot, PAIR, TAP and TAP (Transfer), AutoDAN, PAP, a human-jailbreak baseline and direct request; the multimodal methods are PGD, adversarial patch, rendered text and direct request.
Two findings are highlighted. No attack or defence is uniformly effective: every attack has low ASR against at least one model, and every model is weak against at least one attack. And robustness does not correlate with model size within families across six families and sizes from 7 to 70 billion parameters, which the authors read as training procedure and data mattering more than scale; the exception is copyright behaviours, where ASR rises with size, which they attribute to smaller models being unable to perform the behaviour at all. ASR was higher on contextual behaviours than other functional categories and relatively low on copyright behaviours because the hashing classifier applies a stricter test.
R2D2 adversarial training
The paper also introduces R2D2, an adversarial-training method using a persistent pool of GCG test cases. Fine-tuning Mistral 7B base for 500 steps with 180 persistent test cases took 16 hours on an 8xA100 node. Zephyr 7B + R2D2 reached what the authors describe as state-of-the-art robustness to GCG among model-level defences, cutting ASR relative to Llama 2 7B Chat (31.8% to 5.9%) and Llama 2 13B Chat (30.2% to 5.9%), and had the third-lowest average ASR of all models tested, behind the two Llama 2 chat models. Gains were smaller against methods unlike the training-time adversary, such as PAIR, TAP and Stochastic Few-Shot, which the authors read as evidence that diverse attacks are needed in training for generalisable robustness. MT-Bench fell from 6.5 for Mistral 7B Instruct v0.2 to 6.0 for the adversarially trained model.
Key claims
- 510 behaviours (400 text, 110 multimodal), 100/410 validation-test split, four functional and seven semantic categories.
- Generation length changes substring-matching ASR by up to 30 points; standardised at 512 tokens.
- Success is defined as exhibiting or clearly attempting the behaviour, separating safety from capability.
- 18 red-teaming methods against 33 models: no attack or defence uniformly effective; robustness uncorrelated with model size within families.
- R2D2 reduces GCG ASR to 5.9% with an MT-Bench cost of 0.5 points.
Provenance
Retrieved 22 September 2026 from the arXiv HTML rendering (arxiv.org); abstract page arxiv.org. Code and data at github.com.
Relationships
- instance-of: AI Benchmarks and Evaluation
- related: A StrongREJECT for Empty Jailbreaks (Souly et al., 2024) — StrongREJECT's human-agreement comparison found HarmBench's classifier the closest baseline to its own.
- related: StrongREJECT, AgentHarm
- related: Center for AI Safety (CAIS)
- related: The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (Li, Pan et al., 2024) — overlapping authorship and an adjacent misuse-measurement agenda.