"Cooperative Inverse Reinforcement Learning" is a NeurIPS 2016 paper by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell (UC Berkeley). It gives value alignment a formal definition as a game-theoretic problem.
The paper opens on Norbert Wiener (1960): "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively… we had better be quite sure that the purpose put into the machine is the purpose which we really desire." Its own framing of why this is hard: "Humans are prone to mis-stating their objectives."
The formalism
A CIRL game is "a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is."
Two features distinguish it from classical inverse reinforcement learning. The reward is shared — the robot is not inferring a reward in order to imitate, but is scored on the human's actual reward. And the human is a participant rather than a passive demonstrator, able to act with the robot's learning in mind.
The consequence the authors emphasize is behavioural: "In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment." Alignment-relevant behaviour — asking, demonstrating deliberately, signalling — falls out of the game structure rather than being engineered in.
Results
Reduction to a POMDP (Theorem 1). For any CIRL game with state space S and reward space Θ, there exists a single-actor POMDP with hidden state space of size |S|·|Θ| such that any policy pair in the game has a policy achieving the same discounted return. The reduction works because "the private information is limited to H's initial observation of θ."
Belief as sufficient statistic (Corollary 1). There is an optimal policy pair depending only on the current state and the robot's belief about θ. This is what licenses treating the robot's posterior over the human's objective as the object of interest — the same quantity that drives the shutdown result in The Off-Switch Game.
The complexity gain is substantial: solving the POMDP is "exponentially lower than the NEXP-hard bound" obtained by reducing a CIRL game to a general Dec-POMDP.
Optimality in isolation is suboptimal. Modelling apprenticeship learning as a two-phase CIRL game — a learning phase in which both act, then a deployment phase in which the robot maximizes reward unsupervised — the authors show that "classic IRL falls out as the best-response policy for R under the assumption that the human's policy is 'demonstration by expert' (DBE), i.e., acting optimally in isolation as if no robot exists." But that pair is not optimal: "even if the robot expects expert behavior, demonstrating expert behavior is not the best way to teach that algorithm."
This is the paper's sharpest practical claim. The standard assumption underlying learning-from-demonstration — that the human should just do the task well — is provably not the best available human policy, because good demonstration and good teaching are different objectives. The authors give an approximate algorithm for the human's best response when the robot runs IRL under linear reward assumptions, and confirm the gap empirically on an example game.
Relationships
- related: The Off-Switch Game (Hadfield-Menell, Dragan, Abbeel & Russell, 2016) — same authors and year; supplies the objective-uncertainty framework the shutdown result depends on
- supports: Corrigibility — the formal setting in which uncertainty over the human objective is defined
- related: AI Alignment, RLHF (Reinforcement Learning from Human Feedback) — an alternative formalization of learning objectives from humans, preference comparisons rather than a shared-reward game
- related: Stuart Russell, Instrumental Convergence