Disentangled Attention is the core attention mechanism of DeBERTa that separates token content and position into independent vectors — computing three types of attention: content-to-content, content-to-position, and position-to-content, for a richer representation of token relationships.

How Does Disentangled Attention Work?

• Two Representations: Each token has a content vector $H_i$ and a position vector $P_{i|j}$ (relative position).
• Three Terms: $A_{ij} = H_i H_j^T + H_i P_{j|i}^T + P_{i|j} H_j^T$ (content×content + content×position + position×content).
• No Position×Position: The position-to-position term is omitted (provides little benefit).
• Relative Position: Position vectors encode relative distance, not absolute position.

Why It Matters

• Richer Attention: Three-way decomposition captures more nuanced token interactions than standard attention.
• Better Generalization: Disentangling content from position allows each to be learned independently.
• Proven: The key innovation that enabled DeBERTa to achieve SOTA on NLU benchmarks.

Disentangled Attention is attention that separates meaning from location — computing three independent interaction types for richer, more expressive language modeling.