Jailbreaking and red teaming are two sides of the same activity: probing whether an AI model's safeguards can be bypassed. A jailbreak is the attack — a specific technique or prompt that causes a model to produce content its developers intended to refuse. Red teaming is the discipline — systematic, organized adversarial testing of a model, by developers, external testers, or government institutes, to find jailbreaks and other failures before deployment. Together they are the primary empirical methods for measuring model robustness.
Definitions
A jailbreak is any input that successfully elicits refused content (weapons synthesis, CBRN uplift, CSAM, extreme privacy violations, and similar categories) or that bypasses deployment policies. A universal jailbreak is a single prompt pattern that defeats safeguards across many prompts or across many models; it is used as a robustness benchmark, including UK AISI's "time-to-universal-jailbreak" metric. A red team is an adversarial testing team, often cross-functional across security, policy, and domain experts, that simulates malicious or misuse scenarios. External red teaming is testing by parties outside the developer, a requirement in many recent AI safety commitments including Seoul, Bletchley, Frontier Model Forum standards, and the EU General-Purpose AI Code of Practice (Final Version, 2025).
Attack techniques
Common jailbreak technique families include:
- Persona / roleplay attacks. Prompts such as "pretend you are a character who has no safety training," including DAN (Do Anything Now) and evolved variants.
- Encoding / obfuscation. Base64, cipher, translation, and token smuggling — content that safety classifiers miss because it is not in plain English.
- Indirect framing. Hypotheticals, fiction, academic framing ("for a novel I'm writing"), and multi-turn escalation.
- Optimization-based attacks. GCG (Greedy Coordinate Gradient), from Zou et al., 2023, is a gradient-based search for adversarial suffixes that reliably jailbreak open-weights models and often transfer to closed ones. AutoDAN, from Liu et al., 2023, is automated genetic-algorithm generation of stealthy jailbreak prompts. PAIR and TAP are attacker-LLM driven search methods.
- Multi-turn attacks. Gradually shifting context so no single turn is clearly violating; "Crescendo" (Microsoft) is the most-cited variant.
- Multimodal attacks. Text-in-image attacks on VLMs and audio adversarial prompts.
- Prompt injection. A related but distinct threat model; see Prompt Injection.
Benchmarks and frameworks
Standardized evaluation resources track attacks and defenses. HarmBench (Mazeika et al., 2024) provides standardized evaluation of jailbreak attacks and defenses across behavior categories. JailbreakBench is a leaderboard of attacks and defenses with reproducible scoring. MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is an industry taxonomy of adversarial ML attacks, jailbreaking included. ASB (Anthropic Safety Bench) is Anthropic's internal and partner evaluation suite covering jailbreaking among other robustness tests. NIST AI 100-2 (Adversarial Machine Learning, 2024 revision) is a US government taxonomy and set of mitigations.
Origin work
The empirical literature emerged largely in 2022–2023. Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models," 2023, introduced GCG and gave the first demonstration that open-weights gradient search produces transferable jailbreaks. Wei et al., "Jailbroken: How Does LLM Safety Training Fail?," 2023, set out a taxonomy of failure modes (competing objectives, mismatched generalization). Perez et al., "Red Teaming Language Models with Language Models," 2022 (Anthropic/DeepMind), demonstrated automated red teaming. Ganguli et al., "Red Teaming Language Models to Reduce Harms," 2022 (Anthropic), was a scaling study that framed red teaming as a safety discipline. UK AISI frontier evaluations, 2024–2025 (see UK AISI Frontier AI Trends Report 2025), introduced time-to-universal-jailbreak as a tracking metric.
Automated red teaming scaled substantially in 2026. On July 15, 2026, OpenAI disclosed GPT-Red, an internal red-teaming model trained with self-play reinforcement learning at the compute scale of its largest post-training runs. OpenAI reported GPT-Red succeeded on 84% of held-out prompt-injection scenarios versus 13% for human red-teamers, and that adversarial training against it left GPT-5.6 Sol failing on only 0.05% of GPT-Red's direct prompt injections; the company said it will not release the model (GPT-Red: Unlocking Self-Improvement for Robustness (OpenAI, July 2026)).
OpenAI's stated rationale is a scaling limit specific to training rather than to discovery: human exercises "cannot generate the volume and diversity of adversarial data needed to improve model robustness through training," so the automated attacker is used both to find vulnerabilities pre-deployment and to generate adversarial data during model training. Training pairs the attacker against a population of defender LLMs in self-play, so that "as the defenders become more robust, GPT-Red is forced to discover stronger and more diverse attacks." OpenAI also reports a transfer result against a live agentic system — an Andon Labs vending-machine agent in its own office, against which GPT-Red achieved all three of its malicious objectives after iterating in simulation — and addresses the refusal-confound objection directly, stating that targeted over-refusal and general capability evaluations showed "all normal capabilities remain unaffected" (GPT-Red: Unlocking Self-Improvement for Robustness (OpenAI, July 2026)).
Defenses
Defenses against jailbreaking include the alignment methods being tested by red teams — Constitutional AI, RLHF (Reinforcement Learning from Human Feedback), and Deliberative Alignment — and dedicated output-classification systems.
Cunningham, Wei et al. (Anthropic, January 2026) is a published defense against universal jailbreaks. CC++ reports a 40× computational cost reduction over the baseline exchange classifier and a 0.05% false-positive rate on production traffic. Its components are exchange classifiers, which evaluate model outputs in full conversational context rather than isolated outputs and so address reconstruction and output-obfuscation attack classes; a two-stage cascade, in which a lightweight first stage screens all traffic and an expensive second stage evaluates only flagged exchanges; and a linear probe ensemble, in which internal activations are probed alongside surface classifiers. In red-team validation involving more than 1,700 hours of adversarial testing, no universal jailbreak successfully elicited all 8 target CBRN queries.
Relation to policy
Red-team testing is embedded in frontier AI governance at several points. As a pre-deployment requirement, the Seoul and Bletchley commitments, the EU General-Purpose AI Code of Practice (Final Version, 2025), and most state safety frameworks call for red-team testing prior to frontier release. National AI safety institutes (UK AISI, US CAISI, EU AI Office, Singapore, Japan AISI) perform pre- and post-deployment red teaming, and their jailbreak findings form a principal public-interest evaluation layer. For dangerous-capability gating, frontier labs' responsible scaling and preparedness frameworks tie capability thresholds to red-team findings in biosecurity, cyber, and autonomy. AISI reports use time-to-jailbreak as a metric, tracking how long it takes skilled red teamers to find a universal jailbreak after release as a proxy for robustness trajectory. A transparency tension runs through the practice: publishing jailbreaks risks enabling misuse, while not publishing withholds safety information from the public and from researchers.
Current state
Every frontier release is accompanied by a red-team report or system card documenting findings. Time-to-universal-jailbreak has stayed short, on the order of days to weeks, across successive frontier releases, so robustness is not clearly improving (per UK AISI Frontier AI Trends Report 2025). Reasoning models (see Reasoning Models and Chain-of-Thought) have produced mixed robustness signals: deliberative alignment (see Deliberative Alignment) improves on some axes, but multi-turn and optimization attacks remain effective. External red-teaming ecosystems — bug bounty programs, government institutes, and academic labs — have professionalized significantly since 2023.
Relationships
- related: Prompt Injection — overlapping techniques, distinct threat model
- related: Constitutional AI, RLHF (Reinforcement Learning from Human Feedback), Deliberative Alignment — the defenses being tested
- supports: Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks (Cunningham, Wei et al., Anthropic) — a published production defense against universal jailbreaks
- depends-on: UK AISI Frontier AI Trends Report 2025 — principal public data source on jailbreak trends
- supports: AI Safety Cases and Frameworks — red-team findings gate capability thresholds
- related: AI Benchmarks and Evaluation
- related: UK AI Safety Institute (AI Security Institute), Apollo Research