"Deep Reinforcement Learning from Human Preferences" is a NeurIPS 2017 paper by Paul F. Christiano (OpenAI), Jan Leike (DeepMind), Tom B. Brown, Miljan Martic (DeepMind), Shane Legg (DeepMind), and Dario Amodei (OpenAI). It is the method paper behind RLHF — the technique that, applied to language models five years later, became the industry-standard alignment method.
The problem
The paper's framing is reward specification, not language. Scaling reinforcement learning has succeeded "in domains that have a well-specified reward function," but "many tasks involve goals that are complex, poorly-defined, or hard to specify."
Its worked example is a robot asked to clean a table or scramble an egg: "It's not clear how to construct a suitable reward function, which will need to be a function of the robot's sensors. We could try to design a simple reward function that approximately captures the intended behavior, but this will often result in behavior that optimizes our reward function without actually satisfying our preferences." The authors connect this directly to alignment rather than to engineering convenience: "This difficulty underlies recent concerns about misalignment between our values and the objectives of our RL systems," citing Bostrom (2014), Russell (2016), and Amodei et al. (2016) — the last being Concrete Problems in AI Safety.
Four desiderata are stated for a solution to sequential decision problems lacking a well-specified reward function. It should enable solving "tasks for which we can only recognize the desired behavior, but not necessarily demonstrate it"; allow agents "to be taught by non-expert users"; scale to large problems; and be "economical with user feedback."
The first of these is the conceptual core: it separates recognizing good behavior from producing it, which is what makes the method applicable where demonstrations are unavailable.
Method
The algorithm "fits a reward function to the human's preferences while simultaneously training a policy to optimize the current predicted reward function" — the two running together rather than in sequence.
The human is asked "to compare short video clips of the agent's behavior, rather than to supply an absolute numerical score." The paper's justification is empirical and practical: "We found comparisons to be easier for humans to provide in some domains, while being equally useful for learning human preferences." Comparing short clips "is nearly as fast as comparing individual states" while carrying more information.
On clip length the authors report a calibration finding: "for short clips it took human raters a while just to understand the situation, while for longer clips the evaluation time was a roughly linear function of the clip length. We tried to choose the shortest clip length for which the evaluation time was linear." In Atari, longer clips were often easier to compare "because they provide more context than single frames."
Results
Evaluation covered two domains: Atari games in the Arcade Learning Environment, and robotics tasks in the MuJoCo physics simulator.
Recovering known tasks without the reward function. "A small amount of feedback from a non-expert human, ranging from fifteen minutes to five hours, suffices to learn most of the original RL tasks even when the reward function is not observable," with feedback covering less than 1% of the agent's interactions with the environment.
Learning behaviors with no available reward function. Novel behaviors including "performing a backflip or driving with the flow of traffic" were learned "from about an hour of feedback — even though it is unclear how to hand-engineer a reward function that would incentivize them." The paper claims these "are considerably more complex than any which have been previously learned from human feedback."
The efficiency argument
The discussion states the result as a ratio rather than a benchmark: "Agent-environment interactions are often radically cheaper than human interaction. We show that by learning a separate reward model using supervised learning, it is possible to reduce the interaction complexity by roughly 3 orders of magnitude."
The authors then draw a conclusion that reads against their own method's future direction: the result shows "that we are already hitting diminishing returns on further sample-complexity improvements because the cost of compute is already comparable" to the cost of human feedback. The bottleneck, on this account, had already shifted away from feedback efficiency by 2017.
Significance for later work
The three components that became standard in language-model RLHF are all present: a learned reward model standing in for an unavailable true reward, preference comparisons rather than absolute scores as the human interface, and joint training of reward model and policy. What changed by InstructGPT (2022) was the domain and the addition of a supervised fine-tuning stage, not the core architecture.
The paper also anticipates the failure mode that later motivated much alignment research, in its statement of the problem: optimizing an approximate reward function "will often result in behavior that optimizes our reward function without actually satisfying our preferences." See Reward Hacking.
Relationships
- depends-on: RLHF (Reinforcement Learning from Human Feedback) — the founding method paper for the technique
- related: Training Language Models to Follow Instructions with Human Feedback (InstructGPT) — applies the same reward-model-plus-comparison architecture to instruction-following language models
- related: Concrete Problems in AI Safety — cited as the statement of the misalignment concern the method addresses
- related: Paul Christiano, Jan Leike, Dario Amodei, Reward Hacking, AI Alignment