Weak-to-strong generalization is a research direction in AI alignment that studies whether a less capable supervisor can train a more capable model to perform at the stronger model's own level, rather than being limited to the supervisor's weaker performance. It was introduced by the OpenAI Superalignment team in the December 2023 paper "Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision," whose lead authors include Collin Burns and Pavel Izmailov, with Jan Leike and Ilya Sutskever among the co-authors (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
The analogy
The motivating problem is that aligning superhuman models will require humans to supervise systems much more capable than themselves — for example, to judge whether a million lines of model-written code are safe, or whether a complex argument is sound. Existing alignment methods such as reinforcement learning from human feedback rely on humans being able to evaluate model behavior, an assumption that breaks down once models exceed human ability on the task being supervised (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
Because superhuman models are not yet available to experiment on, the paper proposes an empirical stand-in: use a weak model as a proxy for the human supervisor and a strong model as a proxy for the future superhuman system. The weak model generates labels, the strong model is fine-tuned on those labels, and the question is how much of the strong model's latent capability the weak supervision recovers (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
Findings
Using model pairs across a span of capability — for example a GPT-2-level model supervising a GPT-4-level model — the authors found that naively fine-tuning the strong model on the weak model's labels recovers a substantial portion, but not all, of the performance gap between the two. They report this in terms of performance gap recovered (PGR): the share of the difference between weak and strong performance that the weakly supervised strong model attains. On a set of natural-language-processing tasks, naive fine-tuning recovered roughly half of the gap, and a simple auxiliary confidence loss raised recovery to around 80%, though results were weaker on reward-modeling and chess-puzzle settings (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
The authors frame the result as encouraging evidence that weak supervision can elicit more than the supervisor knows, while cautioning that the methods do not close the gap and that the proxy setup has important disanalogies to the real superalignment problem — for example, future models may imitate human errors in ways current strong models do not imitate weak-model errors. The paper was released alongside an OpenAI grants program funding external research on the problem (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
Position in alignment research
Weak-to-strong generalization is one of the empirical programs associated with the broader effort to supervise systems beyond human ability, alongside Scalable Oversight and Automated Alignment Research. It is frequently cited as a concrete experimental setting for the superalignment agenda and is referenced in connection with debate, recursive reward modeling, and other oversight techniques. It is an empirical analogy rather than a solution: its proponents present it as a tractable way to make measurable progress, and skeptics note that the proxy may not capture the failure modes that matter most for genuinely superhuman systems (Weak-to-Strong Generalization (Burns et al., OpenAI, 2023)).
Relationships
- depends-on: Scalable Oversight (a proposed empirical handle on the same supervision problem)
- related: Automated Alignment Research, AI Alignment, RLHF (Reinforcement Learning from Human Feedback), Recursive Self-Improvement (RSI)
- related: Collin Burns, Jan Leike
- instance-of: AI Alignment (a specific research program within alignment)