positional encoding rope sinusoidal
**Positional Encoding in Transformers** is the **mechanism that injects sequence order information into the position-agnostic attention computation — because self-attention treats its input as an unordered set, positional encodings are essential for the model to distinguish "the cat sat on the mat" from "the mat sat on the cat," with different encoding strategies (sinusoidal, learned, RoPE, ALiBi) offering different tradeoffs in extrapolation ability, computational cost, and representation quality**.
**Why Position Information Is Needed**
Self-attention computes Attention(Q,K,V) = softmax(QK^T/√d)V. This computation is permutation-equivariant — shuffling the input sequence produces the same shuffle in the output. Without position information, the model cannot distinguish word order, making it useless for language (and most sequential data).
**Encoding Strategies**
**Absolute Sinusoidal (Vaswani 2017)**:
- PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
- Each position gets a unique vector added to the token embedding.
- Fixed (not learned). The sinusoidal pattern ensures that relative positions correspond to linear transformations, theoretically enabling generalization beyond training length.
- Limitation: In practice, extrapolation beyond training length is poor.
**Learned Absolute Embeddings**:
- A learnable embedding matrix of shape (max_len, d_model). Position p gets embedding E[p] added to the token embedding.
- Used in BERT, GPT-2. Simple and effective within trained length.
- Cannot extrapolate: position 1025 has no embedding if max_len=1024.
**Rotary Position Embedding (RoPE)**:
- Applies position-dependent rotation to query and key vectors: f(x, p) = R(p)·x, where R(p) is a rotation matrix parameterized by position p.
- The dot product between rotated queries and keys naturally captures relative position: f(q, m)^T · f(k, n) depends on (m-n), the relative position difference.
- Benefits: encodes relative position without explicit relative position computation. Natural extension mechanism via interpolation (NTK-aware, YaRN).
- Used in: LLaMA, GPT-NeoX, Mistral, Qwen, and virtually all modern open-source LLMs.
**ALiBi (Attention with Linear Biases)**:
- No position encoding on embeddings at all. Instead, add a static linear bias to attention scores: bias(i,j) = -m × |i-j|, where m is a head-specific slope.
- The bias penalizes attention to distant tokens proportionally to distance. Different heads use different slopes (geometric sequence), capturing multi-scale dependencies.
- Excellent extrapolation: trains on 1K context, works at 2K+ without modification.
- Used in BLOOM, MPT.
**Comparison**
| Method | Type | Extrapolation | Parameters | Notable Users |
|--------|------|--------------|------------|---------------|
| Sinusoidal | Absolute | Poor | 0 | Original Transformer |
| Learned | Absolute | None | max_len × d | BERT, GPT-2 |
| RoPE | Relative (implicit) | Good (with interpolation) | 0 | LLaMA, Mistral |
| ALiBi | Relative (bias) | Excellent | 0 | BLOOM, MPT |
Positional Encoding is **the information-theoretic bridge between the unordered world of attention and the ordered world of language** — the mechanism whose design determines how well a Transformer can represent sequential structure and, critically, how far beyond its training context the model can generalize.