relative position bias

**Relative Position Bias** is a **position encoding method that adds a learnable bias to attention logits based on the relative distance between query and key positions** — directly encoding "how far apart" two tokens are, rather than their absolute positions. **How Does Relative Position Bias Work?** - **Bias Table**: A learnable matrix $B in mathbb{R}^{(2M-1) imes (2M-1)}$ indexed by relative position $(i-j)$. - **Addition**: $ ext{Attention}(Q, K) = ext{softmax}(QK^T / sqrt{d} + B)$. - **Per-Head**: Different attention heads can have different relative position biases. - **Used In**: Swin Transformer, T5, DeBERTa. **Why It Matters** - **Translation Invariance**: The same relative distance gets the same bias regardless of absolute position. - **Extrapolation**: Can generalize to longer sequences than seen during training (with appropriate handling). - **SOTA**: T5's relative position bias is among the most effective position encodings for NLP. **Relative Position Bias** is **distance-based attention adjustment** — telling the model how to weight attention based on how far apart tokens are, not where they are.