disentangled attention

**Disentangled Attention** is the **core attention mechanism of DeBERTa that separates token content and position into independent vectors** — computing three types of attention: content-to-content, content-to-position, and position-to-content, for a richer representation of token relationships. **How Does Disentangled Attention Work?** - **Two Representations**: Each token has a content vector $H_i$ and a position vector $P_{i|j}$ (relative position). - **Three Terms**: $A_{ij} = H_i H_j^T + H_i P_{j|i}^T + P_{i|j} H_j^T$ (content×content + content×position + position×content). - **No Position×Position**: The position-to-position term is omitted (provides little benefit). - **Relative Position**: Position vectors encode relative distance, not absolute position. **Why It Matters** - **Richer Attention**: Three-way decomposition captures more nuanced token interactions than standard attention. - **Better Generalization**: Disentangling content from position allows each to be learned independently. - **Proven**: The key innovation that enabled DeBERTa to achieve SOTA on NLU benchmarks. **Disentangled Attention** is **attention that separates meaning from location** — computing three independent interaction types for richer, more expressive language modeling.