positional heads

**Positional heads** is the **attention heads whose behavior is dominated by relative or absolute positional relationships between tokens** - they provide structured position-aware routing that other circuits rely on. **What Is Positional heads?** - **Definition**: Heads show strong preference for fixed positional offsets or position classes. - **Role**: Encode ordering and distance information for downstream computations. - **Variants**: Includes previous-token, next-token, and long-range offset-focused patterns. - **Detection**: Observed via relative-position attention histograms and ablation impact. **Why Positional heads Matters** - **Sequence Structure**: Position-aware routing is necessary for order-sensitive language behavior. - **Circuit Foundation**: Many semantic and syntactic circuits build on positional primitives. - **Generalization**: Robust position handling supports long-context behavior quality. - **Failure Debugging**: Positional drift can explain context-length degradation and misalignment. - **Architecture Study**: Useful for comparing positional-encoding schemes across models. **How It Is Used in Practice** - **Offset Profiling**: Quantify attention preference by relative token distance. - **Long-Context Tests**: Evaluate positional-head stability as sequence length grows. - **Ablation**: Remove candidate heads to measure order-sensitivity degradation. Positional heads is **a key positional information channel inside transformer attention** - positional heads are essential infrastructure for reliable sequence-order reasoning in language models.