"Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision" is a December 2023 paper from OpenAI's Superalignment team by Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu.
The problem and the analogy
The paper starts from a dependency in current alignment practice. RLHF works by reinforcing behaviours human evaluators rate highly, which "is very effective when human evaluators" can judge the behaviour. The concern is what happens when they cannot: "future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models."
Because superhuman models do not yet exist, the paper substitutes an analogy that can be run today: "can weak model supervision elicit the full capabilities of a much stronger model?" A weak model stands in for the human, a strong model for the future system. The experiments cover natural language processing, chess, and reward-modeling tasks.
This is the paper's methodological contribution as much as its empirical one — it converts a problem usually discussed prospectively into one with measurable results.
The metric
Performance gap recovered (PGR) is defined against three quantities: weak performance, weak-to-strong performance, and the strong ceiling — "the strong model applying its full capabilities to the task."
PGR = (weak-to-strong − weak) / (strong ceiling − weak)
PGR is 1 under perfect weak-to-strong generalization and 0 if the supervised strong model does no better than its weak supervisor. It measures "the fraction of the performance gap… that we can recover with weak supervision."
Findings
Weak-to-strong generalization is real. "When we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors" — the strong model does not merely imitate the weak labels, including their errors, but exceeds them.
But naive finetuning is not enough. "We are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work." This is the paper's warning, and it is directed at the method currently in use rather than at a hypothetical one.
Simple methods help substantially. "When finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks." The confidence loss encourages the strong model to be confident in its own predictions rather than deferring to the weak labels — that is, to rely on what it already knows where the supervisor is unreliable.
The conclusion is about tractability. "Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models."
Relationships
- supports: Scalable Oversight — supplies an experimental setup for the problem, converting it from prospective to measurable
- contradicts (partial): RLHF (Reinforcement Learning from Human Feedback) — argues the method "may scale poorly to superhuman models without further work"
- related: Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017) — the technique whose scaling limit this paper probes
- related: Jan Leike, Ilya Sutskever, Collin Burns, OpenAI, AI Alignment