This is the standard mechanistic-interpretability account of how in-context learning works in transformer language models. Published on the Transformer Circuits Thread on March 8, 2022, its core research contributors are Catherine Olsson, Nelson Elhage, and Neel Nanda, with a 26-author list from Anthropic including Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, to whom correspondence is addressed. It builds directly on the same group's "A Mathematical Framework for Transformer Circuits" (2021), which introduced induction heads.
The paper is a cumulative case, not a single finding, and the authors are explicit that it does not settle the question: the hypothesis "is a very broad hypothesis and much of our evidence is indirect, but nevertheless we believe that all the lines of evidence together make a relatively strong, though not conclusive, case."
What an induction head is
An induction head is not a single attention head but a circuit of two working across different layers. The first copies information from the previous token into each token; this lets the second attend to tokens by what preceded them rather than by their own content. The second head — the induction head proper — searches back through the sequence for a previous occurrence of the current token A, attends to whatever followed it, B, and raises the probability of emitting B. The pattern is …[A][B]…[A] → [B].
The paper stresses that this is an algorithm rather than a lookup: "induction heads are implementing a simple algorithm, and are not memorizing a fixed table of n-gram statistics. The rule [A][B] … [A] → [B] applies regardless of what A and B are." The name is by analogy to inductive reasoning — inferring that if A was followed by B earlier in the context, it is more likely to be followed by B again — with the induction running over the context rather than over the training data.
The phase change
The paper's most-cited empirical result is a discontinuity early in training. Within a window of roughly 2.5 to 5 billion tokens — for large models, about 1–2% of the way through training — four things happen together: the in-context learning score improves sharply and then stays roughly constant for the remainder of training; induction heads form; the loss curve shows a small "bump," a period of visibly steeper improvement than the segments on either side; and the model's trajectory through the space of per-token losses changes direction abruptly. Before the window there is less than 0.15 nats of in-context learning. The authors call this "the phase change," offering ice melting as the analogy, and note that the one-layer model is the exception that fits: induction heads cannot form in it, and in-context learning does not improve.
The measurement method is worth recording because it is reused elsewhere. The authors collect the log-likelihoods each model snapshot assigns to a fixed set of 10,000 random tokens, each drawn from a different example sequence, assemble these into a "per-token loss vector," and apply principal component analysis, tracing training trajectories through the first two components. The technique traces to Erhan et al. (2010).
Six arguments of unequal strength
The paper's structure is six lines of evidence, and the authors supply their own table grading each against two sub-claims — that induction heads contribute some in-context learning, and that they contribute the majority of it — across three model classes.
| Sub-claim | Small attention-only | Small with MLPs | Large models |
|---|---|---|---|
| Contributes some | Strong, causal | Strong, causal | Medium, correlational and mechanistic |
| Contributes majority | Strong, causal | Medium, causal | Medium, correlational |
The arguments:
- Macroscopic co-occurrence. Models undergo the phase change described above, in which induction heads form and in-context learning improves at the same time, observed across dozens of models of different sizes trained on different datasets.
- Macroscopic co-perturbation. Changing the architecture in a way that shifts whether and when induction heads can form shifts the in-context learning improvement "in a precisely matching way."
- Direct ablation. Knocking out induction heads at test time in small models greatly decreases in-context learning.
- Generality by example. Although induction heads are defined narrowly in terms of copying literal sequences, the same heads are observed implementing more sophisticated in-context learning, including translation and abstract pattern matching.
- Mechanistic plausibility. For small models the mechanism can be explained and shown to contribute to in-context learning, and the mechanism itself suggests how it could be repurposed for more general in-context learning.
- Continuity from small to large. Behaviors and data relating to both induction heads and in-context learning vary smoothly from small to large models, so the simplest explanation is that the mechanism is the same.
The authors state the limitation plainly: "the larger the models get, the harder it is to establish that induction heads account for the actual majority of in-context learning. For large models with MLP's, we must therefore rely on mainly correlational evidence, which could be confounded." They explore alternative hypotheses in Arguments 1 and 6. A summary that reduced the paper to "induction heads cause in-context learning" would overstate a claim the authors themselves present as converging evidence of differing strength.
Beyond literal copying
Argument 4 is the part most often abbreviated away. Heads that meet the narrow definition — copying literal repeated sequences — are also observed performing translation and abstract pattern matching, that is, completing [A*][B*] … [A] → [B] where the earlier tokens are similar rather than identical. This is what supports the paper's broader claim: if the same circuit generalizes from literal repetition to abstract completion, it becomes plausible that it accounts for a large fraction of in-context learning rather than a narrow slice of it.
Safety framing
The paper situates itself in the mechanistic-interpretability programme, whose motivating theory it states as reverse engineering neural networks in order to have grounded confidence in their safety, and identifies three bearings.
On phase changes: "If neural network behavior discontinuously changes from one scale to the next, this makes it more challenging for researchers and society to prepare for future problems." On in-context learning: it means model behavior "can in some sense 'change' during inference, without further training," and even under the weaker reading that in-context learning locates an already-learned behavior rather than acquiring a new one, "the behavior could be a surprising and unwanted off-distribution generalization." On mesa-optimization: against the concern that in-context learning might be implemented by an internally-developed optimization algorithm, the paper reports that the mechanism is induction heads, at least in small models, and states "we did not observe any evidence of mesa-optimizers."
The authors also propose the phase change as a "Rosetta stone" linking mechanistic interpretability, learning dynamics, and scaling-law-style empirical properties, on the grounds that it is concrete, reproducible in small models, confined to a small segment of training, and tied to a capability of independent interest.
Provenance
Published on transformer-circuits.pub on March 8, 2022, under Anthropic affiliation. Fetched and verified August 27, 2026. The byline, affiliation, and date were read from the document's own author block rather than imported from citation convention. The paper's metadata carries no DOI, and a second template metadata block reads "Not published yet" alongside the March 8, 2022 publication date. The page carries reader-contributed replication notes in a "Comments & Replications" section.
Relationships
- supports: In-Context Learning — the standard mechanistic account of the capability, which that page previously recorded as absent.
- related: Mechanistic Interpretability — an early result of the programme, and one of the field's most-cited.
- related: Emergent capabilities — the phase change is one of the disputed instances of emergence, and this paper supplies a mechanism for it in one case.
- related: Anthropic — the publisher.
- related: Scaling Laws — the paper proposes its phase change as a link between mechanistic interpretability, learning dynamics, and scaling-law-style empirical work.
- related: Reasoning Models and Chain-of-Thought, AI Control — the safety framing turns on behavior changing at inference time without further training.