Induction Heads are the specific two-layer attention head circuits in transformer models that implement pattern matching by searching for previously-seen context and predicting the token that followed it — identified as the mechanistic foundation of in-context learning and representing one of the most significant discoveries in mechanistic interpretability research.
What Are Induction Heads?
- Definition: A circuit consisting of two attention heads (one in layer 1, one in layer 2) that together implement the algorithm: "Search the current context for a previous occurrence of the current token, then predict the token that followed it."
- Pattern: Implements the rule [A][B]...[A] → predict [B]. If the model saw "Harry Potter" earlier, and now sees "Harry," it dramatically increases probability of "Potter."
- Discovery: Identified by Olsson et al. (Anthropic, 2022) in "In-context Learning and Induction Heads" — one of the first complete mechanistic accounts of a transformer capability.
- Universality: Induction heads form in virtually every transformer model trained on sequential prediction tasks — from 1-layer toy models to GPT-style production models.
Why Induction Heads Matter
- In-Context Learning Mechanism: Induction heads are the primary mechanism behind in-context learning (few-shot prompting) — demonstrating that this capability has a specific, identifiable mechanical implementation rather than being mysterious emergent behavior.
- Phase Transition: Induction heads form during a sudden phase transition in training — a specific training step where loss drops sharply and in-context learning ability appears. This phase transition is one of the clearest examples of capability emergence in neural network training.
- Universality: The fact that the same circuit forms independently in models of very different sizes and architectures demonstrates that transformers learn canonical algorithms — supporting the hope that interpretability findings generalize.
- Mechanistic Interpretability Proof of Concept: Induction heads demonstrated that it is possible to identify, understand, and formally describe a real computational mechanism inside a transformer — validating the mechanistic interpretability research program.
How Induction Heads Work — The Mechanism
The Two-Head Circuit:
Head 1 — Previous Token Head (layer L₁):
- Attends to the previous token in the sequence at each position.
- Copies information from position [t-1] to position [t].
- Creates a "shifted-by-one" key: K[t] contains information about token at position [t-1].
Head 2 — Induction Head (layer L₂, L₂ > L₁):
- Queries: "What token am I currently at?"
- Keys: Use output of Head 1 (shifted-by-one information).
- Match: Find positions where K[j] matches Q[t] — i.e., find where the token that preceded position j matches the current token at position t.
- Value: Copy the value at position j (the token that actually follows the matched position).
- Result: Attend to position [j] where token[j-1] = token[t], and predict token[j+1].
In-Context Few-Shot Learning:
- When given examples (input₁, output₁), (input₂, output₂), ..., (input_test, ?):
- Induction heads match input_test to previous inputs in context and copy the associated outputs.
- This is mechanistically why few-shot prompting works — the model's attention circuitry pattern-matches to provided examples and copies their associated outputs.
The Phase Transition
During transformer training, a clear phase transition occurs at a specific training step:
- Before: Model relies on unigram statistics (predict most common next tokens).
- During phase transition: Induction heads form in ~1 training step of rapid loss decrease.
- After: Model in-context learning improves dramatically; model tracks patterns within context window.
Evidence: Ablating the attention heads that form during the phase transition restores the pre-transition loss — confirming these heads causally produce the capability.
Induction Head Variants
- Fuzzy Induction Heads: Match not on exact token identity but on semantic similarity — predict tokens that follow semantically similar contexts.
- Multi-step Induction: Generalized circuits that implement longer-range pattern completion.
- Translation Heads: In multilingual models, heads that map between languages using analogous induction-like pattern matching.
Implications for AI Safety
- Emergent Capability Mechanism: Phase transitions in AI capability may generally correspond to the formation of specific circuits — not mystical emergence but identifiable mechanical changes.
- In-Context Learning = Circuit: The fact that ICL is implemented by identifiable attention heads means we can potentially modify, amplify, or suppress in-context learning through targeted intervention.
- Research Template: The induction head discovery established the methodological template for identifying circuits: activation patching → attention pattern analysis → weight inspection → formal algorithm reconstruction.
Induction heads are the Rosetta Stone of mechanistic interpretability — the first complete, formal account of a transformer capability that validated the entire research program of understanding neural networks as reverse-engineered algorithms rather than inscrutable black boxes, demonstrating that even seemingly mysterious capabilities like in-context learning have precise, understandable mechanical implementations.