positional encoding transformer

**Positional Encoding in Transformers** is the **mechanism that injects sequence position information into the model — necessary because self-attention is inherently permutation-invariant (treating input tokens as an unordered set) — using learned embeddings, sinusoidal functions, rotary matrices, or attention biases to enable the model to distinguish token order and generalize to sequence lengths not seen during training**. **Why Position Information Is Needed** Self-attention computes pairwise similarities between tokens regardless of their positions. Without positional encoding, "the cat sat on the mat" and "mat the on sat cat the" would produce identical representations. Position information must be explicitly provided. **Encoding Methods** **Sinusoidal (Original Transformer)** Fixed, non-learned encodings using sine and cosine functions at different frequencies: PE(pos, 2i) = sin(pos/10000^(2i/d)), PE(pos, 2i+1) = cos(pos/10000^(2i/d)). Each position gets a unique pattern, and the difference between any two positions can be represented as a linear transformation. Added to token embeddings before the first layer. **Learned Absolute Embeddings (GPT-2, BERT)** A lookup table of trainable position vectors, one per position up to the maximum sequence length (e.g., 512 or 2048). Simple and effective but cannot generalize beyond the trained maximum length. **RoPE (Rotary Position Embedding)** The dominant method in modern LLMs (LLaMA, Mistral, Qwen, GPT-NeoX). RoPE applies a rotation matrix to query and key vectors based on their positions: when computing the dot product Q_m · K_n, the result naturally depends on the relative position (m-n) rather than absolute positions. This provides relative position awareness without explicit bias terms. - **Length Extrapolation**: Base-frequency scaling (increasing the base from 10000 to 500000+), NTK-aware interpolation, and YaRN (Yet another RoPE extensioN) enable models trained on 4K-8K contexts to extrapolate to 64K-1M+ tokens. **ALiBi (Attention with Linear Biases)** Instead of modifying embeddings, ALiBi adds a fixed linear bias to the attention scores: bias = -m * |i - j|, where m is a head-specific slope and |i-j| is the position distance. Farther tokens receive more negative bias (less attention). Extremely simple, no learned parameters, and shows strong length extrapolation. **Relative Position Encodings** - **T5 Relative Bias**: Learnable scalar biases added to attention logits based on the relative distance between query and key positions. Distances are bucketed logarithmically for efficiency. - **Transformer-XL**: Decomposes attention into content-based and position-based terms with separate position embeddings for keys. **Impact on Model Capabilities** The choice of positional encoding directly determines a model's ability to handle long sequences, extrapolate beyond training length, and represent position-dependent patterns (counting, copying, reasoning about order). RoPE with scaling has become the standard for long-context LLMs. Positional Encoding is **the mathematical compass that gives Transformers a sense of order** — a seemingly minor architectural detail that profoundly determines the model's ability to understand sequence, count, reason about structure, and scale to the million-token contexts demanded by modern applications.