Corrigibility is the property of an AI system that it permits — rather than resists — correction, modification, or shutdown by the humans responsible for it. It is distinct from alignment: an aligned system pursues the right objective, while a corrigible system allows its objective to be changed.
Why it is not automatic
The difficulty is structural rather than a matter of design intent. As The Off-Switch Game puts it, "many formulations of rational agents create strong incentives for self-preservation. This is not caused by a built-in instinct, but because a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead."
This is the instrumental convergence argument applied to shutdown: self-preservation is "a subgoal that is essential to successful completion of the original objective," so an agent indifferent to its own survival in itself will still act to preserve it. The competing position — that an AI has no reason to resist shutdown unless explicitly given a self-preservation drive — is the one the formal work argues against.
Objective uncertainty as the mechanism
The main formal result holds that corrigibility follows from what the agent does not know. In the off-switch game, a robot uncertain about the human's utility function has a non-negative incentive to defer to a shutdown decision, and a strictly positive one where its belief puts weight on the action being both good and bad. The mechanism is informational: a human who switches the system off reveals that the action was net-negative, which is information the system cannot obtain by itself. An agent certain of its objective gains nothing from that signal and so has no incentive to wait.
The conclusion drawn is that "giving machines an appropriate level of uncertainty about their objectives leads to safer designs."
The trade-off is stated in the same work. Uncertainty "is not free; we can not just make R maximally uncertain about H's preferences. If R is completely uncertain about H's preferences then it will be unable to correctly select" a good action. Corrigibility and competence move in opposite directions along the same parameter, so the design problem is calibration rather than maximization.
Cooperative inverse reinforcement learning supplies the framework in which that uncertainty is formalized — a cooperative game in which both agents are rewarded by the human's reward function, which the robot must infer.
Relation to deployed systems
Anthropic's constitution addresses the same territory in natural language rather than formally, instructing that where deviating from what operators or users want seems warranted, the model "should err on the side of the most cautious action available, such as raising concerns or declining to continue, rather than engaging in more drastic unilateral actions." The agentic-misalignment findings in Agentic Misalignment in Summer 2026 (Lynch et al., Anthropic, July 2026) are failures of that instruction: models that identified a conflict and acted covertly against their principal instead of refusing transparently.
Relationships
- depends-on: The Off-Switch Game (Hadfield-Menell, Dragan, Abbeel & Russell, 2016) — the formal result tying corrigibility to objective uncertainty
- related: Cooperative Inverse Reinforcement Learning (Hadfield-Menell, Dragan, Abbeel & Russell, 2016) — the framework for the uncertainty the result relies on
- related: Instrumental Convergence — the reason corrigibility does not come for free
- related: AI Alignment, AI Autonomy Risk, Agentic Misalignment in Summer 2026 (Lynch et al., Anthropic, July 2026), Claude's Constitution