positional encoding
**Positional Encoding Methods** — Positional encodings inject sequence order information into transformer architectures that are inherently permutation-invariant, enabling models to distinguish token positions and capture sequential structure.
**Sinusoidal Positional Encoding** — The original transformer used fixed sinusoidal functions at different frequencies to encode absolute positions. Each dimension uses sine or cosine functions with geometrically increasing wavelengths, creating unique position signatures. This approach generalizes to unseen sequence lengths through its continuous nature and encodes relative positions through linear transformations of the encoding vectors. However, fixed encodings cannot adapt to task-specific positional patterns.
**Learned Absolute Embeddings** — BERT and GPT models learn position embedding vectors as trainable parameters, one per position up to a maximum sequence length. These embeddings are added to token embeddings before processing. Learned embeddings can capture task-specific positional patterns but are limited to the maximum length seen during training. Extrapolation beyond training lengths typically degrades performance significantly without additional techniques.
**Rotary Position Embeddings (RoPE)** — RoPE encodes positions by rotating query and key vectors in 2D subspaces at position-dependent angles. This elegant formulation naturally encodes relative positions through the rotation angle difference, while being compatible with linear attention approximations. RoPE has become the dominant positional encoding for modern large language models including LLaMA, PaLM, and their derivatives. NTK-aware scaling and YaRN extend RoPE to longer contexts by modifying the frequency base or applying interpolation strategies.
**Relative Position Methods** — ALiBi (Attention with Linear Biases) adds position-dependent linear biases directly to attention scores, penalizing distant token pairs. This simple approach requires no additional parameters and extrapolates well to longer sequences than seen during training. T5's relative position bias learns scalar biases for bucketed relative distances, sharing biases across attention heads. Relative encodings generally outperform absolute methods for length generalization.
**Positional encoding design has emerged as a critical factor in transformer capability, particularly for length generalization, with modern methods like RoPE and ALiBi enabling models to process sequences far beyond their training context while maintaining coherent positional reasoning.**