rotary position embedding

**Rotary Position Embedding (RoPE)** is **a positional encoding method that encodes token position as rotation angles in complex plane, applying multiplicative rotation to query/key vectors — achieving superior extrapolation beyond training sequence length compared to absolute positional embeddings**. **Mathematical Foundation:** - **Complex Representation**: encoding position m as e^(im*θ) with frequency θ varying by dimension — contrasts with absolute embeddings adding fixed vectors - **2D Rotation Matrix**: applying rotation to q and k vectors: [[cos(m*θ), -sin(m*θ)], [sin(m*θ), cos(m*θ)]] — preserves dot product magnitude across rotations - **Frequency Schedule**: θ_d = 10000^(-2d/D) with d ∈ [0, D/2) varying frequency per dimension — lower frequencies for positional differences, higher for fine details - **Dimension Pairing**: each 2D rotation applies to consecutive dimension pairs, reducing complexity from O(D²) to O(D) — RoPE paper reports 85% faster computation **Practical Advantages Over Absolute Embeddings:** - **Length Extrapolation**: training on 2048 tokens enables inference on 4096+ tokens with <2% perplexity degradation — absolute embeddings show 40-60% degradation - **Relative Position Focus**: dot product (q_m)·(k_n) = |q||k|cos(θ(m-n)) depends only on relative position m-n — perfectly captures translation invariance - **Reduced Parameters**: no learnable position embeddings table (saves 2048×4096=8.4M params for 4K context) — critical for efficient fine-tuning - **Interpretability**: rotation angles directly correspond to position differences — explainable compared to black-box learned embeddings **Implementation in Transformers:** - **Llama 2 Architecture**: uses RoPE as default with base frequency 10000 and dimension 128 — inference on up to 4096 tokens - **GPT-Neo**: original implementation with linear frequency schedule θ_d = base^(-2d/D) supporting length interpolation - **YaLM-100B**: integrates RoPE with ALiBi positional biases, achieving 16K context window — Yandex foundational model - **Qwen LLM**: extends RoPE with dynamic frequency scaling for variable-length training up to 32K tokens **Extension Mechanisms:** - **Position Interpolation**: increasing base frequency multiplier β when extrapolating to new length — enables 4K→32K without retraining with only 1% perplexity increase - **Frequency Scaling**: modifying base frequency to lower values (e.g., 10000→100000) shifts rotation rates for longer sequences - **Alien Attention**: hybrid combining RoPE with Ali attention biases for improved long-context performance - **Coupled Positional Encoding**: using RoPE jointly with absolute embeddings in hybrid approach — CodeLlama uses this for 16K context **Rotary Position Embedding is the state-of-the-art positional encoding — enabling transformers to achieve superior length extrapolation and efficient long-context inference across Llama, Qwen, and PaLM models.**