"Deliberative Alignment: Reasoning Enables Safer Language Models" is a December 2024 paper by Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Amelia Glaese, and colleagues at OpenAI. It is the method paper for Deliberative Alignment, the technique used to align the o-series models.
The diagnosis
The paper starts from the limits of the standard pipeline. Models safety-trained with supervised fine-tuning and RLHF "still exhibit safety shortcomings: they can be tricked into revealing harmful content, often refuse legitimate requests, and remain vulnerable to jailbreak attacks."
Its alternative is to give the model the specification itself and train it to reason over it — "directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering."
The motivating observation is empirical rather than theoretical: "given access to our actual safety policies, o1 models are often able to correctly reason over how to respond to potentially unsafe prompts." The naive version of that — putting all specification text in the system prompt at deployment — is rejected on cost: "such an approach comes with a clear latency cost."
Method
Two stages, "integrating process- and outcome-based supervision."
Stage 1 — supervised fine-tuning on specification-referencing chains of thought. The dataset is built by context distillation: present a helpfulness-only model — "an o-type model trained only for helpfulness (i.e. trained without any safety-relevant data)" — with the safety specifications in its system prompt, generate completions, "and then strip away the system prompts to form the final dataset." The result is (prompt, CoT, output) triples whose reasoning cites the specifications, with no human-written chains of thought or answers anywhere in the pipeline.
Stage 2 — outcome-based reinforcement learning, which "refines the CoT safety reasoning."
The authors characterize the division of labour: "process-supervision provides a strong prior, and… outcome-based RL refines the CoT safety reasoning."
Results
The headline claim is a Pareto improvement rather than a single-metric gain. The o1 models "advance the Pareto frontier of refusing to answer malicious jailbreak prompts (from StrongREJECT) and not over-refusing benign prompts (from XSTest), compared to GPT-4o and other state-of-the-art LLMs," with error bars from 1,000 bootstrap trials.
This matters because jailbreak robustness and overrefusal normally trade against each other — a model made harder to jailbreak typically refuses more legitimate requests. Moving both at once is the paper's central empirical claim. It also reports improved out-of-distribution generalization.
The framing offered for why reasoning helps: "chain-of-thought reasoning can serve to leverage test-time compute to improve safety behavior, ultimately training LLMs to be 'right for the right reasons'." The authors argue the approach yields alignment that is "more scalable, trustworthy, and interpretable," since the reasoning that produced a refusal is inspectable rather than implicit in the weights.
Relation to other approaches
The method is the counterpart to Anthropic's Constitutional AI in relying on a written specification and model-generated training data rather than human labels, but differs in where the specification is applied: Constitutional AI uses principles to generate critique-and-revision training signal, while deliberative alignment trains the model to invoke the specification text at inference time within its chain of thought.
It is also the mechanism named in OpenAI's Model Spec methodology post as a partial remedy for generalization differing from intent — "not a complete solution."
Relationships
- depends-on: Deliberative Alignment — the method paper
- related: OpenAI Model Spec — the specification text the method trains models to reason over
- related: Constitutional AI: Harmlessness from AI Feedback — the closest counterpart approach, differing in where the written specification enters
- related: RLHF (Reinforcement Learning from Human Feedback), Reasoning Models and Chain-of-Thought, Distillation, OpenAI