positional encoding transformer

**Positional Encodings in Transformers** are the **mechanisms that inject sequence order information into the attention mechanism — which is inherently permutation-invariant — enabling the model to distinguish between tokens at different positions and generalize to sequence lengths beyond those seen during training, with modern approaches like RoPE and ALiBi replacing the original sinusoidal encodings**. **Why Position Information Is Needed** Self-attention computes Q·Kᵀ between all token pairs — the operation treats the token sequence as an unordered set. Without positional information, the sentences "dog bites man" and "man bites dog" produce identical attention patterns. Positional encodings break this symmetry. **Encoding Methods** - **Sinusoidal (Vaswani et al., 2017)**: Fixed positional vectors using sine and cosine functions at different frequencies: PE(pos, 2i) = sin(pos/10000^(2i/d)), PE(pos, 2i+1) = cos(pos/10000^(2i/d)). Added to token embeddings before the first attention layer. Theoretical length generalization through frequency composition, but limited in practice. - **Learned Absolute Embeddings**: A learnable embedding table with one vector per position (BERT, GPT-2). Simple but rigidly tied to maximum training length — cannot extrapolate beyond the training context window. - **Relative Position Bias (T5, Transformer-XL)**: Instead of encoding absolute position, inject a learned bias based on the relative distance (i-j) between query token i and key token j directly into the attention score. Better generalization to longer sequences because the model learns distance relationships rather than absolute positions. - **RoPE (Rotary Position Embedding)**: Applied in LLaMA, Mistral, Qwen, and most modern LLMs. Encodes position by rotating the query and key vectors in 2D subspaces: pairs of dimensions are rotated by position-dependent angles. The dot product Q·Kᵀ then naturally encodes relative position through the angle difference. RoPE provides: - Relative position awareness through rotation angle difference - Decaying inter-token dependency with increasing distance - Flexible length extrapolation via frequency scaling (NTK-aware, YaRN, Dynamic NTK) - **ALiBi (Attention with Linear Biases)**: Subtracts a linear penalty proportional to token distance directly from attention scores: attention_score -= m·|i-j|, where m is a head-specific slope. No learned parameters. Excellent length extrapolation; simpler than RoPE but less expressive. **Context Length Extension** RoPE-based models can extend their context window beyond training length through: - **Position Interpolation (PI)**: Scale all positions into the training range (e.g., map 0-8K to 0-4K). Requires fine-tuning. - **NTK-Aware Scaling**: Modify the rotation frequencies's base value to spread position information across more dimensions. Better preservation of local position resolution. - **YaRN**: Combines NTK scaling with temperature adjustment and attention scaling, achieving strong long-context performance with minimal fine-tuning. Positional Encodings are **the hidden mechanism that gives transformers their sense of order and distance** — a seemingly minor architectural detail whose choice directly determines whether a language model can handle 4K or 1M+ token contexts.