Induction Heads are the specific two-layer attention head circuits in transformer models that implement pattern matching by searching for previously-seen context and predicting the token that followed it — identified as the mechanistic foundation of in-context learning and representing one of the most significant discoveries in mechanistic interpretability research.

What Are Induction Heads?

• Definition: A circuit consisting of two attention heads (one in layer 1, one in layer 2) that together implement the algorithm: "Search the current context for a previous occurrence of the current token, then predict the token that followed it."
• Pattern: Implements the rule [A][B]...[A] → predict [B]. If the model saw "Harry Potter" earlier, and now sees "Harry," it dramatically increases probability of "Potter."
• Discovery: Identified by Olsson et al. (Anthropic, 2022) in "In-context Learning and Induction Heads" — one of the first complete mechanistic accounts of a transformer capability.
• Universality: Induction heads form in virtually every transformer model trained on sequential prediction tasks — from 1-layer toy models to GPT-style production models.

Why Induction Heads Matter

• In-Context Learning Mechanism: Induction heads are the primary mechanism behind in-context learning (few-shot prompting) — demonstrating that this capability has a specific, identifiable mechanical implementation rather than being mysterious emergent behavior.
• Phase Transition: Induction heads form during a sudden phase transition in training — a specific training step where loss drops sharply and in-context learning ability appears. This phase transition is one of the clearest examples of capability emergence in neural network training.
• Universality: The fact that the same circuit forms independently in models of very different sizes and architectures demonstrates that transformers learn canonical algorithms — supporting the hope that interpretability findings generalize.
• Mechanistic Interpretability Proof of Concept: Induction heads demonstrated that it is possible to identify, understand, and formally describe a real computational mechanism inside a transformer — validating the mechanistic interpretability research program.

How Induction Heads Work — The Mechanism

The Two-Head Circuit:

Head 1 — Previous Token Head (layer L₁):

• Attends to the previous token in the sequence at each position.
• Copies information from position [t-1] to position [t].
• Creates a "shifted-by-one" key: K[t] contains information about token at position [t-1].

Head 2 — Induction Head (layer L₂, L₂ > L₁):

• Queries: "What token am I currently at?"
• Keys: Use output of Head 1 (shifted-by-one information).
• Match: Find positions where K[j] matches Q[t] — i.e., find where the token that preceded position j matches the current token at position t.
• Value: Copy the value at position j (the token that actually follows the matched position).
• Result: Attend to position [j] where token[j-1] = token[t], and predict token[j+1].

In-Context Few-Shot Learning:

• When given examples (input₁, output₁), (input₂, output₂), ..., (input_test, ?):
• Induction heads match input_test to previous inputs in context and copy the associated outputs.
• This is mechanistically why few-shot prompting works — the model's attention circuitry pattern-matches to provided examples and copies their associated outputs.

The Phase Transition

During transformer training, a clear phase transition occurs at a specific training step:

• Before: Model relies on unigram statistics (predict most common next tokens).
• During phase transition: Induction heads form in ~1 training step of rapid loss decrease.
• After: Model in-context learning improves dramatically; model tracks patterns within context window.

Evidence: Ablating the attention heads that form during the phase transition restores the pre-transition loss — confirming these heads causally produce the capability.

Induction Head Variants

• Fuzzy Induction Heads: Match not on exact token identity but on semantic similarity — predict tokens that follow semantically similar contexts.
• Multi-step Induction: Generalized circuits that implement longer-range pattern completion.
• Translation Heads: In multilingual models, heads that map between languages using analogous induction-like pattern matching.

Implications for AI Safety

• Emergent Capability Mechanism: Phase transitions in AI capability may generally correspond to the formation of specific circuits — not mystical emergence but identifiable mechanical changes.
• In-Context Learning = Circuit: The fact that ICL is implemented by identifiable attention heads means we can potentially modify, amplify, or suppress in-context learning through targeted intervention.
• Research Template: The induction head discovery established the methodological template for identifying circuits: activation patching → attention pattern analysis → weight inspection → formal algorithm reconstruction.

Induction heads are the Rosetta Stone of mechanistic interpretability — the first complete, formal account of a transformer capability that validated the entire research program of understanding neural networks as reverse-engineered algorithms rather than inscrutable black boxes, demonstrating that even seemingly mysterious capabilities like in-context learning have precise, understandable mechanical implementations.