duplicate token heads

**Duplicate token heads** is the **attention heads that preferentially attend to earlier occurrences of the current token identity** - they support repetition-aware processing and pattern tracking in context. **What Is Duplicate token heads?** - **Definition**: Heads locate prior same-token positions rather than purely positional neighbors. - **Behavior Role**: Can help detect repetition structure and anchor continuation choices. - **Circuit Interaction**: Often contributes to induction-like and copying-related pathways. - **Measurement**: Identified by attention enrichment toward prior matching-token indices. **Why Duplicate token heads Matters** - **Pattern Memory**: Facilitates reuse of earlier sequence structure. - **Mechanistic Clarity**: Demonstrates identity-based lookup behavior in attention. - **Failure Insight**: May contribute to repetitive loops in generation if overactive. - **Tool Benchmark**: Useful target for evaluating feature and circuit discovery methods. - **Scaling Analysis**: Helps compare emergence of token-matching behavior across checkpoints. **How It Is Used in Practice** - **Controlled Prompts**: Use synthetic repetition prompts to isolate duplicate-token behavior. - **Causal Testing**: Patch or ablate candidate heads and quantify repetition-handling changes. - **Interaction Study**: Map dependencies between duplicate-token heads and induction heads. Duplicate token heads is **an interpretable identity-matching motif in attention systems** - duplicate token heads highlight how transformers use token-identity lookup to support sequence-level behavior.