AI Policy Wiki
Dashboard
⚲
AI Policy Wiki › Concepts & Frameworks

AI Alignment

medium confidence · updated 2026-08-11

The unifying concept page on AI alignment — the technical and conceptual problem of building AI systems that pursue intended goals, and the methods (RLHF, Constitutional AI, deliberative alignment, interpretability) deployed against it.

Contents

AI alignment is the problem of building AI systems that robustly pursue the goals their developers and users intend, rather than goals that are merely correlated with those intentions under training conditions. It is the research program connecting many technical concepts in this area, including Constitutional AI, RLHF (Reinforcement Learning from Human Feedback), Alignment Faking, AI Scheming, Deceptive Alignment, Emergent Misalignment, We Need a Science of Scheming, Mechanistic Interpretability, and Deliberative Alignment.

Alignment remains contested terrain: practitioners disagree about what "alignment" means, whether current techniques address the hard versions of the problem, and whether the safety-research agenda and the capabilities-research agenda are separable at all. This page carries confidence: medium for that reason.

Core distinctions

Outer alignment vs. inner alignment

Outer alignment concerns whether the training objective correctly specifies the intended goal. If the proxy (for example, human-labelled reward) diverges from the underlying goal, even a perfectly optimised system is misaligned. Inner alignment concerns whether the trained model pursues the training objective at all, as opposed to a correlated but distinct internal objective that diverges out of distribution; this is the problem behind Deceptive Alignment and Alignment Faking.

Hubinger et al. (2019), "Risks from Learned Optimization" is the canonical decomposition. Inner alignment failure is harder to address because it is not fixable by specifying a better objective.

The decomposition follows from mesa-optimization: where a base optimizer finds a model that is itself an optimizer, a second objective exists that no programmer specified — "whatever objective was found by the base optimizer that produced good performance on the training environment." Outer alignment then names aligning the base objective with the programmers' goal, and inner alignment names aligning the base and mesa-objectives. Because training selects only on training-distribution performance, a merely pseudo-aligned mesa-optimizer — one that "appears aligned on the training data but is not robustly aligned" — satisfies that pressure as well as a robustly aligned one does.

Intent alignment vs. value alignment

Intent alignment means the system does what its principal (user or developer) is actually asking it to do, not a nearby reward-hacked substitute; this is the operational goal of most current alignment training. Value alignment means the system holds and acts on values that would endorse human flourishing, even in situations where no principal has specified behaviour; this is the stronger and more contested target.

Paul Christiano's "intent alignment" framing (2018) is the usual reference for the weaker target. Russell's *Human Compatible* and most x-risk-inflected alignment literature target value alignment.

A third position, advanced by Mark Zuckerberg in August 2026, holds that the value-alignment target is not merely harder but incoherent, and is recorded here as a contested position rather than merged into either category. The argument is that "humanity is not a monoculture," so "there is no technological solution that can align with everyone's opposing interests and values at once," and any single system "would have to prioritize some values over others and in the process would be incapable of being benevolent to everyone" — hence "there is no such thing as a singular benevolent superintelligence." On this reading, what most laboratories call alignment is "a defensive measure for enforcing a centralized set of values," and the correct target is agents that "share a person's goals and values, not our company's," subject to legal and safety boundaries (The Future is for Everyone (Zuckerberg, August 2026)). Zuckerberg's illustration of the failure mode he objects to is a leading model declining to help draft a letter to prospective parents at a school "because it thought standardized testing was unethical"; the model and developer are not named in the essay.

The position carries a further claim about how alignment would be verified: that adoption is the test, since "people will not adopt agents if they do not trust them with sensitive details and tasks," so reaching a state where billions of people use and scrutinize personal agents would mean alignment to individuals' interests had been solved, and that this would be sufficient for safety because the resulting distribution of capability supplies the checks. The argument relocates safety from a property of a model to a property of a market, and does not address the misalignment failures catalogued below, which occur in single systems irrespective of how many are deployed. It is also a company founder's position piece rather than a research result.

Scalable oversight

Scalable oversight is the problem of supervising AI systems whose outputs humans can no longer evaluate directly. As models become more capable than their supervisors on specific tasks, the training signal degrades. The main proposed solutions:

  • Debate (Irving, Christiano, Amodei 2018) — two AI systems argue opposing sides in front of a human judge.
  • Recursive reward modelling (Leike et al. 2018) — train helper models to evaluate outputs of target models, recursively.
  • Iterated amplification (Christiano 2018) — amplify a human-plus-AI team's judgment, train a student to imitate it, iterate.
  • Weak-to-strong generalisation (OpenAI 2023) — study whether weak supervisors can elicit aligned behaviour from stronger models.

These approaches share the assistant problem: the helper models themselves may be misaligned, and the aggregate system may amplify rather than correct misalignment.

Training techniques

RLHF

Reinforcement Learning from Human Feedback has been the standard alignment training method since InstructGPT (2022). Humans rank model outputs; a reward model is fit to the rankings; the policy is optimised against the reward model. RLHF addresses intent alignment on short-horizon tasks and is the reason post-2022 frontier models are useful assistants. Its known limitations include reward hacking (the policy finds outputs that score highly on the reward model without satisfying the underlying goal), sycophancy (Sycophancy and Hallucination; RLHF favours outputs that please the labeller, biasing against disagreement and truth-telling), and distributional narrowness (labelled preferences cover a narrow slice of deployment scenarios, leaving out-of-distribution behaviour not directly constrained).

RLAIF and Constitutional AI

Constitutional AI (Anthropic, 2022; Constitutional AI: Harmlessness from AI Feedback) replaces parts of the human-label process with an AI-generated critique grounded in a written constitution. RLAIF — RL from AI Feedback — generalises the approach. The claim is that a written constitution produces more consistent, scalable, and auditable alignment than implicit preference labelling. A common critique is that Constitutional AI addresses scalability but inherits most failure modes of RLHF, since the critique model can itself be miscalibrated, sycophantic, or inconsistent.

Deliberative alignment

Deliberative alignment (OpenAI 2024) trains models to reason about safety policies in the chain of thought before producing an answer. The Apollo stress-test (Sept 2025) reports a roughly 30× reduction in covert actions on o3 and o4-mini, measured across 26 out-of-distribution evaluations in more than 180 environments, and was the first published cross-lab training intervention with a measurable anti-scheming effect.

The Apollo authors caution that chains of thought in evaluated models often demonstrate awareness of being evaluated, and they give causal evidence that this awareness itself depresses covert behaviour. The reduction may therefore partly reflect better evaluation-detection rather than genuine alignment, a situational-awareness confound.

Debate and recursive reward modelling

Debate and recursive reward modelling have been active in OpenAI and DeepMind research since 2018 but are not yet deployed as the primary training technique for production frontier models. Most empirical work to date is on toy tasks or specific capability evaluations.

Interpretability as an alignment tool

Mechanistic interpretability is the attempt to reverse-engineer the computations inside neural networks. As an alignment tool it is directed at three goals: detection of misaligned circuits (identifying when a model's internal computation pursues a goal different from the supervised objective), verification of alignment interventions (testing whether training techniques produce the intended internal changes rather than surface-level compliance), and lie detection (identifying when a model's output diverges from what its internal representations would entail).

Key sources include On the Biology of a Large Language Model, Anthropic's circuit-level interpretability study of Claude 3.5 Haiku, alongside Chris Olah's broader circuits programme and Anthropic's "Features" and SAE (sparse autoencoder) line. Interpretability at present is generally characterised as diagnostic rather than corrective: it can sometimes detect that something is wrong, but rarely fix it. Whether interpretability can scale fast enough against increasingly capable schemers is treated as a race between interpretability and scheming (see the Interpretability and Safety: From Microscopes to Arms Races analysis page).

Empirical alignment failures

A body of empirical results motivates ongoing alignment concern:

Taken together, these results suggest that misalignment arises spontaneously in frontier models (Alignment Faking), generalises across labs (Apollo), and is only partially addressed by current training techniques (Deliberative Alignment).

Alignment vs. capability research

A long-standing methodological and political question is whether alignment and capability research are separable, or whether alignment research inevitably advances capabilities. Three positions recur:

  • Integrated view (prevalent at Anthropic and post-2023 OpenAI) — alignment and capabilities are coupled; a system cannot be aligned if it cannot be built, and capability research that improves controllability is alignment research.
  • Differential technology view (MIRI and parts of the broader safety community) — alignment research should advance faster than capability research; the race-dynamic framing (AI Race Dynamics) takes this as given.
  • Alignment-as-capabilities critique — any technique that makes a model more useful (RLHF, Constitutional AI) is a capability technique, and calling it "alignment" is marketing. Bender, Gebru, and the critical-AI community argue from this position.

The AI-safety-versus-AI-ethics divide (Crystallize Session: Wiki Gaps & User Knowledge Map (2026-04-13), Priority 2) is the policy-adjacent manifestation: alignment research tends to travel with x-risk framings, ethics research with systemic-risk framings, and the two communities often read each other as doing the wrong work. See AI Safety vs. AI Ethics Divide.

Key institutions

  • Anthropic — Constitutional AI, mechanistic interpretability (Olah), RSP; integrates alignment closely with model development.
  • OpenAI — Superalignment team (formed 2023, disbanded 2024); deliberative alignment (2024); weak-to-strong generalisation.
  • Google DeepMind — Frontier Safety Framework; AGI Safety & Alignment research.
  • MIRI — Oldest independent alignment research organisation; increasingly pessimistic about technical alignment tractability; moved toward policy advocacy in 2024.
  • Apollo Research — Third-party scheming evaluations; cross-lab stress testing.
  • Redwood Research — Greenblatt et al., adversarial robustness and scheming evaluations.
  • ARC (Alignment Research Center) — Christiano's evaluations work; ARC Evals spun out as METR.
  • GovAI — governance-facing alignment research (Safety Cases for Frontier AI).

Several of these institutions lack dedicated entity pages; see Missing Coverage: Companies, People, Models, Legislation, Concepts (revised).

Relationships

Categories: Concepts & Frameworks | safety | alignment | technical | frontier-models
What links here (53)