previous token heads

**Previous token heads** is the **attention heads that strongly attend to the immediately preceding token position** - they provide local context routing that supports many higher-level circuits. **What Is Previous token heads?** - **Definition**: Attention pattern is concentrated on token index minus one relative position. - **Functional Use**: Creates short-range context features used by downstream heads. - **Circuit Role**: Often upstream of induction and local-grammar processing mechanisms. - **Detection**: Identified through average attention maps and positional preference metrics. **Why Previous token heads Matters** - **Foundational Routing**: Local token transfer is a building block for many model computations. - **Interpretability Baseline**: Simple positional behavior provides clear mechanistic anchors. - **Composition Insight**: Helps explain how later heads build complex behavior from local signals. - **Error Analysis**: Weak or noisy local routing can degrade syntax and continuation quality. - **Comparative Study**: Useful for scaling analyses across model sizes and architectures. **How It Is Used in Practice** - **Positional Probes**: Measure head attention by relative position across diverse prompts. - **Circuit Mapping**: Trace which later components consume previous-token features. - **Intervention**: Ablate candidate heads and monitor local dependency performance drops. Previous token heads is **a basic but important positional mechanism in transformer attention** - previous token heads are critical primitives for constructing higher-order sequence-processing circuits.