"Concrete Problems in AI Safety" is a 2016 paper by Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané (affiliated with Google Brain, OpenAI, and Stanford), published 21 June 2016 as arXiv:1606.06565. It reframes AI safety as a portfolio of near-term technical research problems concerning accidents and enumerates five such problems that became standard vocabulary in subsequent alignment research.
Authors: Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané (Google Brain / OpenAI / Stanford) Published: 21 June 2016 (arXiv:1606.06565)
Summary of argument
The paper argues that AI safety should be studied not as a philosophical problem about far-future systems, but as a portfolio of concrete, near-term technical research problems concerning accidents, defined as "unintended and harmful behavior that may emerge from poor design of real-world AI systems." It organizes the AI safety research agenda around five problems that remained core vocabulary a decade later.
The five problems
- Avoiding negative side effects — an agent pursuing an objective should not unnecessarily disrupt its environment (the "don't break vases" problem).
- Avoiding reward hacking — preventing agents from gaming the specified reward signal rather than pursuing the intended goal.
- Scalable oversight — efficiently ensuring correct behavior on aspects of the objective too expensive to evaluate frequently (a motivation for later work on RLHF and Constitutional AI).
- Safe exploration — ensuring exploratory actions during learning do not cause catastrophic outcomes.
- Robustness to distributional shift — recognizing and behaving correctly when environments differ from training.
The paper categorizes the problems by origin: wrong objective function (1, 2), objective too expensive to evaluate (3), and undesirable behavior during learning (4, 5).
Key claims
The paper presents accident risk as a technical research problem rather than speculation, grounding each of the five problems in observable failures of existing reinforcement-learning systems. It argues that reward hacking scales with capability, with more capable optimizers more readily discovering reward-gaming strategies, a theme later formalized in AI scheming research. It also argues that scalable oversight is necessary for capable systems, since humans cannot check every decision of a highly capable agent and oversight must therefore be amplified, motivating later work on RLHF, debate, and Constitutional AI.
Influence
The paper is upstream of a large fraction of the modern alignment research program: RLHF, Constitutional AI, mechanistic interpretability, and scheming research all descend from or address these five problems. Two co-authors, Amodei and Christiano, later became central figures at Anthropic and in the alignment research community.
Relationships
- depends-on: AI Safety Frameworks — this paper frames the foundational vocabulary
- supports: AI Autonomy Risk — reward hacking and distributional shift are mechanisms for autonomy failures
- related: RLHF — scalable oversight motivates the RLHF research line
- related: Constitutional AI — CAI is a scalable-oversight technique
- related: AI Scheming — reward hacking taxonomy is a precursor concept
- related: Dario Amodei, Paul Christiano — co-authors