Rotary Position Embedding (RoPE) is a positional encoding method that encodes token position as rotation angles in complex plane, applying multiplicative rotation to query/key vectors — achieving superior extrapolation beyond training sequence length compared to absolute positional embeddings.

Mathematical Foundation:

• Complex Representation: encoding position m as e^(im*θ) with frequency θ varying by dimension — contrasts with absolute embeddings adding fixed vectors
• 2D Rotation Matrix: applying rotation to q and k vectors: [[cos(mθ), -sin(mθ)], [sin(mθ), cos(mθ)]] — preserves dot product magnitude across rotations
• Frequency Schedule: θ_d = 10000^(-2d/D) with d ∈ [0, D/2) varying frequency per dimension — lower frequencies for positional differences, higher for fine details
• Dimension Pairing: each 2D rotation applies to consecutive dimension pairs, reducing complexity from O(D²) to O(D) — RoPE paper reports 85% faster computation

Practical Advantages Over Absolute Embeddings:

• Length Extrapolation: training on 2048 tokens enables inference on 4096+ tokens with <2% perplexity degradation — absolute embeddings show 40-60% degradation
• Relative Position Focus: dot product (q_m)·(k_n) = |q||k|cos(θ(m-n)) depends only on relative position m-n — perfectly captures translation invariance
• Reduced Parameters: no learnable position embeddings table (saves 2048×4096=8.4M params for 4K context) — critical for efficient fine-tuning
• Interpretability: rotation angles directly correspond to position differences — explainable compared to black-box learned embeddings

Implementation in Transformers:

• Llama 2 Architecture: uses RoPE as default with base frequency 10000 and dimension 128 — inference on up to 4096 tokens
• GPT-Neo: original implementation with linear frequency schedule θ_d = base^(-2d/D) supporting length interpolation
• YaLM-100B: integrates RoPE with ALiBi positional biases, achieving 16K context window — Yandex foundational model
• Qwen LLM: extends RoPE with dynamic frequency scaling for variable-length training up to 32K tokens

Extension Mechanisms:

• Position Interpolation: increasing base frequency multiplier β when extrapolating to new length — enables 4K→32K without retraining with only 1% perplexity increase
• Frequency Scaling: modifying base frequency to lower values (e.g., 10000→100000) shifts rotation rates for longer sequences
• Alien Attention: hybrid combining RoPE with Ali attention biases for improved long-context performance
• Coupled Positional Encoding: using RoPE jointly with absolute embeddings in hybrid approach — CodeLlama uses this for 16K context

Rotary Position Embedding is the state-of-the-art positional encoding — enabling transformers to achieve superior length extrapolation and efficient long-context inference across Llama, Qwen, and PaLM models.