The InstructGPT paper is a 2022 OpenAI research paper reporting that fine-tuning GPT-3 with reinforcement learning from human feedback (RLHF) produces models more aligned with user intent than the base model, including a finding that a 1.3B-parameter aligned model is preferred by human evaluators over the 100x-larger 175B GPT-3. It is the canonical paper for RLHF and provided the post-training method that OpenAI later used in ChatGPT.
Authors: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe (OpenAI) Published: 4 March 2022 (arXiv:2203.02155) Commonly known as: the InstructGPT paper
Method
The paper describes a three-step training pipeline:
- Supervised fine-tuning (SFT). Labeler-written demonstrations of desired behavior on prompts (including real OpenAI API prompts) are collected and used to fine-tune GPT-3.
- Reward model training. Human rankings of multiple model outputs are collected, and a reward model is trained to predict human preference.
- RLHF. The SFT model is fine-tuned with reinforcement learning (PPO), using the reward model as the reward signal.
Findings
The paper reports that aligned small models can beat unaligned large ones on human preference: the 1.3B-parameter InstructGPT model was preferred by human evaluators over the 175B-parameter GPT-3 base model. The authors describe this as a surprising result that reframed alignment as capability-enhancing for user-facing tasks. The paper also reports gains in truthfulness on TruthfulQA and reductions in toxic generation. On public capability benchmarks it reports minimal regression, with an accompanying discussion of the "alignment tax." Its core framing is that RLHF is a viable general method for aligning large language models with user intent.
In wiki confidence terms, the preference and truthfulness/toxicity results are held at high confidence and the minimal-capability-regression claim at medium confidence.
Reception and influence
The method described in the paper was the post-training approach behind ChatGPT, released in November 2022, which OpenAI's account ties to the consumer AI uptake that followed. RLHF or a variant has since been adopted across frontier labs including OpenAI, Anthropic, Google DeepMind, Meta, and DeepSeek. RLHF's dependence on large volumes of human labels motivated Anthropic's Constitutional AI / RLAIF approach as a scalable-oversight variant addressing the human-label bottleneck.
Relationships
- supports: RLHF — this is the concept's canonical paper
- depends-on: Concrete Problems in AI Safety — operationalizes scalable oversight
- related: Constitutional AI — CAI is a successor method addressing RLHF's human-label bottleneck
- related: OpenAI, Paul Christiano, Jan Leike
- related: AI Benchmarks and Evaluation — preference ranking as eval