positional encoding
**Positional Encoding for Transformers**
**Why Positional Encoding?**
Transformers have no inherent notion of sequence order. Positional encoding injects position information so the model knows where each token is in the sequence.
**Encoding Methods**
**Sinusoidal Positional Encoding (Original Transformer)**
$$
PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})
$$
$$
PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})
$$
- Fixed, not learned
- Can extrapolate to longer sequences (in theory)
- Added to token embeddings
**Learned Positional Embeddings**
- Trainable embedding for each position
- Used in GPT-2, BERT
- Cannot extrapolate beyond training length
**RoPE (Rotary Position Embedding)**
Used by: Llama, Mistral, Qwen, and most modern models
Key ideas:
- Encodes position in the rotation of query and key vectors
- Relative position naturally emerges from the dot product
- Better length extrapolation than absolute encodings
```python
# Simplified RoPE application
def apply_rope(x, freqs):
# Split into pairs, rotate by position-dependent angle
x_rotated = rotate_half(x) * freqs
return x * torch.cos(freqs) + x_rotated * torch.sin(freqs)
```
**ALiBi (Attention with Linear Biases)**
Used by: MPT, BLOOM
- No position encoding in embeddings
- Subtracts linear bias from attention scores based on distance
- Excellent extrapolation properties
- Simple: $score_{ij} = q_i \cdot k_j - m \cdot |i - j|$
**Comparison**
| Method | Extrapolation | Learning | Modern Use |
|--------|---------------|----------|------------|
| Sinusoidal | Limited | Fixed | Less common |
| Learned | None | Trainable | Legacy |
| RoPE | Good (with scaling) | Fixed | Most popular |
| ALiBi | Excellent | Fixed | Some models |
**Length Extrapolation**
RoPE can be extended with:
- **Linear scaling**: Divide positions by factor
- **NTK-aware scaling**: Adjust frequency base
- **YaRN**: Position interpolation with attention scaling