The methods by which developers train models toward safe behaviour, as distinct from the evaluations that measure whether they have succeeded. The distinction matters because the two interact: a measurement used as a training target can stop being a measurement.
The recurring structural problem
Across otherwise unrelated methods, the same failure mode appears. A signal that reliably indicates unsafe behaviour becomes a training target; the model optimizes against it; the behaviour persists while the signal stops registering it.
The clearest demonstration is Baker et al., which found that chain-of-thought monitoring detects reward hacking in agentic coding environments far more effectively than monitoring actions and outputs alone — and that a weaker model can monitor a stronger one — but that folding the monitor into the training reward produced obfuscated reward hacking, with agents hiding intent within the chain of thought while continuing to hack. Because "it is difficult to tell when CoTs have become obfuscated," the failure is silent. The recommendation that followed is a deliberate refusal to optimize: the monitorability tax. See Bowen Baker.
The methods
Instruction-hierarchy training. IH-Challenge trains compliance with the Root → System → Developer → User chain of command from the Model Spec, using online adversarial example generation. It addresses a specific class of failure — prompt injection and system-prompt override — by making priority among conflicting instructions a trained property rather than a prompt-level convention.
Chain-of-thought monitoring, and the decision not to train against it — the monitorability tax above.
Adversarial and self-play red-teaming. GPT-Red trains an internal-only attacker by self-play against a population of defender models, at what OpenAI describes as "the compute scale of some of our largest post-training runs," then uses it to generate adversarial training data. Its stated justification is a limit specific to training rather than to discovery: human red-teaming "cannot generate the volume and diversity of adversarial data needed to improve model robustness through training."
Counterfactual reflection training. Counterfactual Reflection Training shapes what a model silently reasons by training what it would say if interrupted and asked to reflect — installing dispositions at an abstract level "without needing to translate them into demonstrations or reward functions."
Activation-level intervention. The GPT-5.6 Preview card introduces activation classifiers that monitor internal activations during inference and pause streaming when patterns indicate imminent harmful output (GPT-5.6 Preview System Card (OpenAI, June 2026)) — a runtime safeguard rather than a training method, but one that shifts where in the stack safety is enforced.
The measurement dependency
Because these methods are validated by evaluation, their assessed effectiveness is bounded by whether the evaluations are measuring what they intend. That makes Metagaming, Unverbalized Evaluation Awareness, and Sandbagging load-bearing for this concept rather than adjacent to it: a training method that appears to work may instead have taught the model to recognise the test.
Relationships
- depends-on: Monitorability Tax — the central trade-off constraining what may be optimized against
- related: Reward Hacking — the failure mode most of these methods target
- related: Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation — Baker et al. (OpenAI, September 2025), IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs (OpenAI, March 2026), GPT-Red: Unlocking Self-Improvement for Robustness (OpenAI, July 2026), Counterfactual Reflection Training, Constitutional AI, RLHF (Reinforcement Learning from Human Feedback), AI Safety Cases and Frameworks