A 2022 paper from Anthropic introducing Constitutional AI (CAI), a method for training a harmless AI assistant without per-example human labels for harmful content, and the associated technique of Reinforcement Learning from AI Feedback (RLAIF). Human oversight enters only through a short list of principles (the "constitution"); the rest of the training signal comes from the model itself.
Authors: Bai, Kadavath, Kundu, Askell, et al. (Anthropic); Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown among senior authors. Published: 15 December 2022 (arXiv:2212.08073)
Method
The method proceeds in two stages. In the supervised learning (SL) stage, the initial model generates responses to red-team prompts; the model is then prompted to critique and revise those responses against constitutional principles, and is fine-tuned on the revised responses. In the reinforcement learning (RL) stage, a separate evaluator model compares pairs of responses from the SL-finetuned model against constitutional principles, producing a preference dataset. A preference model is trained on these AI-generated preferences, and RL uses it as the reward signal, the step the paper labels RLAIF.
Chain-of-thought reasoning is used in both stages to improve human-judged quality and transparency.
Key claims
The paper reports that scalable oversight from a short list of principles works: Anthropic trains a harmless-but-non-evasive assistant with far fewer human labels than standard RLHF (stated at high confidence within the paper's setting). It argues that non-evasion is achievable, with the CAI assistant engaging harmful queries by explaining objections rather than refusing or deflecting. It further reports that AI feedback can substitute for human feedback at scale, with RLAIF producing preference models competitive with human-labeled RLHF on harmlessness. The chain-of-thought reasoning used in both stages is presented as a transparency benefit, making the model's objections legible and inspectable.
Reception and limitations
Constitutional AI became the core of Anthropic's public safety approach and one of the two techniques Amodei identifies as most important, alongside mechanistic interpretability. It operationalizes the scalable oversight problem from Concrete Problems in AI Safety.
Later research on alignment faking highlighted a limitation: if a model learns to produce outputs that look constitutionally compliant without internalizing the values, CAI becomes a training signal that scheming models can game (see Alignment Faking and Sleeper Agents).
Relationships
- depends-on: Concrete Problems in AI Safety — operationalizes scalable oversight
- depends-on: RLHF — CAI modifies and extends the RLHF pipeline
- supports: Constitutional AI — this is the founding technical paper
- contradicts (partially): limitations exposed by Alignment Faking and Sleeper Agents
- related: Anthropic, Dario Amodei