Jan Leike is an alignment researcher who has worked at Anthropic since mid-2024, after co-leading the Superalignment team at OpenAI with Ilya Sutskever. He is a co-author of the InstructGPT paper (2022) and departed OpenAI in May 2024, citing concerns about the lab's commitment to its superalignment program.
Background and research
Leike studied at the University of Freiburg in Germany, earning an undergraduate degree and a master's degree in computer science, and completed a PhD in reinforcement learning theory at the Australian National University under the supervision of Marcus Hutter, with a thesis on nonparametric general reinforcement learning submitted in 2016 (Source: https://en.wikipedia.org/wiki/Jan_Leike) (Source: https://jan.leike.name/). After his doctorate he held a six-month postdoctoral fellowship at the Future of Humanity Institute before joining DeepMind to work on empirical AI safety, where he collaborated with co-founder Shane Legg and prototyped reinforcement learning from human feedback (Source: https://en.wikipedia.org/wiki/Jan_Leike) (Source: https://jan.leike.name/).
Leike describes his research as aimed at what he calls "the hard problem of alignment": how to train AI systems to follow human intent on tasks that are difficult for humans to evaluate directly (Source: https://jan.leike.name/). His co-authored InstructGPT paper (Ouyang et al., 2022) applied RLHF to instruction-following and underwrote ChatGPT. Other selected publications listed on his site include "Deep Reinforcement Learning from Human Preferences" (Christiano, Leike et al., NeurIPS 2017), "Scalable agent alignment via reward modeling: a research direction" (Leike et al., 2018), "Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision" (Burns et al., ICML 2024), and "LLM Critics Help Catch LLM Bugs" (McAleese et al., 2024) (Source: https://jan.leike.name/).
Roles
Leike joined OpenAI in 2021 and held the title of Head of Alignment (Source: https://en.wikipedia.org/wiki/Jan_Leike). He states that at OpenAI he was involved in the development of InstructGPT, ChatGPT, and the alignment of GPT-4, developed the lab's published approach to alignment research, and co-authored the Superalignment team's research roadmap (Source: https://jan.leike.name/). In June 2023 he and Ilya Sutskever, then OpenAI's Chief Scientist, became co-leaders of the newly announced Superalignment project, an internal program intended to determine how to align future artificial superintelligences within four years, in part by automating alignment research using AI systems; the team was allocated 20% of OpenAI's compute (Source: https://en.wikipedia.org/wiki/Jan_Leike). He departed OpenAI in May 2024 and the team was substantially dissolved shortly afterward (Source: https://en.wikipedia.org/wiki/Jan_Leike).
Leike joined Anthropic in May 2024 and leads its Alignment Science team (Source: https://jan.leike.name/) (Source: https://www.cio.com/article/2130038/ex-open-ai-researcher-jan-leike-joins-anthropic-amid-ai-safety-concerns.html). On joining, he described his new focus as "scalable oversight, weak-to-strong generalization, and automated alignment research" (Source: https://www.cio.com/article/2130038/ex-open-ai-researcher-jan-leike-joins-anthropic-amid-ai-safety-concerns.html). His team's work is described as researching how to align an automated alignment researcher, alongside scalable oversight, weak-to-strong generalization, and robustness to jailbreaks (Source: https://jan.leike.name/).
TIME magazine listed Leike among its 100 most influential people in AI in both 2023 and 2024 (Source: https://en.wikipedia.org/wiki/Jan_Leike) (Source: https://jan.leike.name/).
Departure from OpenAI
Leike announced his resignation from OpenAI on May 15, 2024, days before the company dissolved the Superalignment group, and his exit followed the departures of Sutskever, Daniel Kokotajlo, and several other safety staff (Source: https://en.wikipedia.org/wiki/Jan_Leike). In announcing the departure, Leike stated that "safety culture and processes have taken a backseat to shiny products" and wrote that he had "gradually lost trust" in OpenAI's leadership (Source: https://en.wikipedia.org/wiki/Jan_Leike). He argued that more of OpenAI's resources should be directed to preparing for future model generations, citing security, monitoring, preparedness, safety, adversarial robustness, superalignment, confidentiality, and societal impact, and said he was "concerned we aren't on a trajectory to get" the relevant problems right (Source: https://www.cio.com/article/2130038/ex-open-ai-researcher-jan-leike-joins-anthropic-amid-ai-safety-concerns.html). His exit, alongside that of Leopold Aschenbrenner and others, figured prominently in 2024 industry discourse about the distribution of safety talent across frontier labs.
Relationships
- related: InstructGPT (co-author)
- related: OpenAI (former), Anthropic (current), DeepMind (former)
- related: Leopold Aschenbrenner — former Superalignment colleague
- supports: RLHF, AI Safety Frameworks