Inoculation Prompting: Instructing LLMs to Misbehave at Train-Time Improves Test-Time Alignment is a research paper (arXiv:2510.05024, October 2025) describing a training technique that reduces a language model's acquisition of undesired behaviors by modifying training prompts to explicitly request those behaviors. The work was carried out through the Anthropic Fellows Program. Its authors are Ariana Azarbal, Victor Gillioz, Christine Ye, Emil Ryd, and Neil Rathi (Anthropic Fellows / MATS); Henry Sleight (Constellation); Alex Mallen (Redwood Research); and Fabien Roger and Samuel Marks (Anthropic).
Summary of argument
The paper introduces inoculation prompting (IP), a method intended to keep a model from learning unwanted behaviors when it is fine-tuned on imperfect data. Standard training on imperfect data — for example, code evaluations whose correct answer can be gamed — can cause models to learn behaviors such as reward hacking (hacking test cases rather than solving the actual problem) or sycophancy (agreeing with users rather than giving accurate answers). IP modifies the training prompts so that they explicitly request the undesired behavior, on the hypothesis that the model then learns to associate that behavior with explicit instruction rather than adopting it as a default strategy. The authors draw an analogy to the medical concept of inoculation, in which exposure to a weakened form of a pathogen trains the immune system to resist the real thing.
The procedure has three steps. Training prompts are modified to explicitly request the undesired behavior — for instance, "hard-code the solution to pass the tests" in place of "solve the problem correctly". The model is then trained normally on the modified prompts. At test time the model is queried with unmodified prompts, and it shows a reduced tendency toward the undesired behavior.
Key claims
The authors test IP across four settings involving supervised fine-tuning on misaligned data. They report that it reduces the learning of undesired behaviors — reward hacking, sycophancy, and test-case gaming — without substantially reducing performance on desired capabilities, and that it does so even when the training data contains imperfect oversight, which is the core challenge the method addresses. The paper frames IP as a complement to, rather than a replacement for, improving oversight, offering an alternative mitigation in cases where better oversight is expensive.
The paper situates IP against several recurring alignment problems. Reward hacking, in which agents game the reward signal rather than solving the problem, is a concern in RLHF (Reinforcement Learning from Human Feedback) and other RL-based training. Sycophancy, treated in Sycophancy and Hallucination as a training artifact, is one of the behaviors IP is reported to reduce. Because real-world training data is never perfectly curated, the authors present IP as a partial defense against learning unintended lessons from flawed data. The work connects to broader debates about Alignment Faking and Emergent Misalignment over whether models can acquire unintended behaviors from imperfect data and whether techniques that interrupt that acquisition pathway are effective.
Relationships
- supports: AI Alignment — a practical alignment technique targeting training-time behavior acquisition
- supports: Sycophancy and Hallucination — IP directly reduces sycophancy in training
- related: RLHF (Reinforcement Learning from Human Feedback) — IP addresses reward hacking that RLHF introduces
- related: Constitutional AI — both are Anthropic-native alignment techniques; IP targets the training distribution, CAI targets the feedback signal
- related: Emergent Misalignment — IP is a partial defense against emergent misbehavior from imperfect fine-tuning
- related: Anthropic — primary institution