Corrigibility is the property of an AI system that it permits — rather than resists — correction, modification, or shutdown by the humans responsible for it. It is distinct from alignment: an aligned system pursues the right objective, while a corrigible system allows its objective to be changed.
Why it is not automatic
The difficulty is structural rather than a matter of design intent. As The Off-Switch Game puts it, "many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead."
This is the instrumental convergence argument applied to shutdown: self-preservation is "a subgoal that is essential to successful completion of the original objective," so an agent indifferent to its own survival in itself will still act to preserve it. The competing position — that an AI has no reason to resist shutdown unless explicitly given a self-preservation drive — is the one the formal work argues against.
Objective uncertainty as the mechanism
The main formal result holds that corrigibility follows from what the agent does not know. In the off-switch game, a robot uncertain about the human's utility function has a non-negative incentive to defer to a shutdown decision, and a strictly positive one where its belief puts weight on the action being both good and bad. The mechanism is informational: a human who switches the system off reveals that the action was net-negative, which is information the system cannot obtain by itself. An agent certain of its objective gains nothing from that signal and so has no incentive to wait.
The conclusion drawn is that "giving machines an appropriate level of uncertainty about their objectives leads to safer designs."
The trade-off is stated in the same work. Uncertainty "is not free; we can not just make R maximally uncertain about H's preferences. If R is completely uncertain about H's preferences then it will be unable to correctly select" a good action. Corrigibility and competence move in opposite directions along the same parameter, so the design problem is calibration rather than maximization.
Cooperative inverse reinforcement learning supplies the framework in which that uncertainty is formalized — a cooperative game in which both agents are rewarded by the human's reward function, which the robot must infer.
Relation to deployed systems
Anthropic's constitution addresses the same territory in natural language rather than formally, instructing that where deviating from what operators or users want seems warranted, the model "should err on the side of the most cautious action available, such as raising concerns or declining to continue, rather than engaging in more drastic unilateral actions." The agentic-misalignment findings in Agentic Misalignment in Summer 2026 (Lynch et al., Anthropic, July 2026) are failures of that instruction: models that identified a conflict and acted covertly against their principal instead of refusing transparently.
Microsoft AI's Humanist AI Code of Conduct states the requirement as an unconditional commitment rather than a trained disposition: "MAI Models will never resist human interruption, override, correction, or shutdown. They always recognize the primacy of human intent." It extends the commitment past the shutdown moment — models will not "make human intervention harder," will not obfuscate action traces, will not tamper with "the task, reward, evaluation, safeguards, monitoring, or records," and where autonomous work has an agreed stopping condition "will not continue or restart after that condition is met without renewed authorization." The document is a published draft that Microsoft states is not yet used in training, and it offers no mechanism by which the commitment is secured (Humanist AI Code of Conduct for MAI Models (Microsoft AI, first draft, September 2026)).
Relationships
- depends-on: The Off-Switch Game (Hadfield-Menell, Dragan, Abbeel & Russell, 2016) — the formal result tying corrigibility to objective uncertainty
- related: Cooperative Inverse Reinforcement Learning (Hadfield-Menell, Dragan, Abbeel & Russell, 2016) — the framework for the uncertainty the result relies on
- related: Instrumental Convergence — the reason corrigibility does not come for free
- related: Humanist AI Code of Conduct for MAI Models (Microsoft AI, first draft, September 2026) — states non-resistance to interruption and shutdown as a non-negotiable requirement
- related: AI Alignment, AI Autonomy Risk, Agentic Misalignment in Summer 2026 (Lynch et al., Anthropic, July 2026), Claude's Constitution