Positional Encoding Methods are the techniques for injecting sequence position information into Transformer models, which otherwise treat input as an unordered set — enabling the model to distinguish token order and capture positional relationships through absolute position embeddings, relative position biases, or rotation-based encodings that generalize to longer sequences than seen during training.

Absolute Positional Encodings:

• Sinusoidal Encoding (Original Transformer): PE(pos, 2i) = sin(pos/10000^(2i/d)), PE(pos, 2i+1) = cos(pos/10000^(2i/d)); deterministic function of position and dimension; different frequencies for different dimensions enable the model to learn to attend by relative position; theoretically allows extrapolation to longer sequences but empirically limited
• Learned Absolute Embeddings: trainable embedding matrix of size max_length × d_model; each position has a learnable vector added to token embeddings; used in BERT, GPT-2; simple and effective but cannot generalize beyond max_length seen during training; requires retraining or interpolation for longer sequences
• Extrapolation Problem: both sinusoidal and learned absolute encodings struggle with sequences longer than training length; attention patterns learned at position 512 don't transfer well to position 2048; motivates relative position methods
• Position Interpolation: linearly interpolates learned position embeddings to extend context; if trained on length L and want length 2L, use embeddings at positions 0, 0.5, 1.0, 1.5, ...; enables 2-4× context extension with minimal fine-tuning

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Positional Encoding — How Transformers Know Order</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">without position info, attention is permutation-invariant — position encoding breaks symmetry</text>

  <!-- 4 methods side by side -->
  <!-- Sinusoidal -->
  <rect x="30" y="70" width="170" height="210" rx="6" fill="#080d14" stroke="#60a5fa" stroke-width="1"/>
  <text x="115" y="90" fill="#93c5fd" font-size="9" font-weight="600" text-anchor="middle">Sinusoidal (2017)</text>
  <path d="M45,120 C60,110 70,130 85,120 C100,110 110,130 125,120 C140,110 150,130 165,120 C180,110 185,130 190,120" fill="none" stroke="#60a5fa" stroke-width="1.5"/>
  <path d="M45,145 C70,130 95,160 120,145 C145,130 170,160 190,145" fill="none" stroke="#60a5fa" stroke-width="1" opacity="0.6"/>
  <text x="115" y="168" fill="#8b98a5" font-size="7.5" text-anchor="middle">PE(pos,2i) = sin(pos/10000^(2i/d))</text>
  <text x="115" y="184" fill="#8b98a5" font-size="7.5" text-anchor="middle">fixed, no parameters</text>
  <text x="115" y="200" fill="#8b98a5" font-size="7.5" text-anchor="middle">absolute position</text>
  <text x="115" y="220" fill="#6b7684" font-size="7" text-anchor="middle">added to input embeddings</text>
  <text x="115" y="240" fill="#6b7684" font-size="7" text-anchor="middle">original Transformer</text>
  <text x="115" y="258" fill="#f87171" font-size="7" text-anchor="middle">no extrapolation beyond</text>
  <text x="115" y="270" fill="#f87171" font-size="7" text-anchor="middle">training length</text>

  <!-- Learned -->
  <rect x="210" y="70" width="170" height="210" rx="6" fill="#080d14" stroke="#34d399" stroke-width="1"/>
  <text x="295" y="90" fill="#6ee7b7" font-size="9" font-weight="600" text-anchor="middle">Learned (2018)</text>
  <!-- Embedding matrix -->
  <rect x="235" y="108" width="120" height="55" rx="3" fill="#0b1220" stroke="#334155" stroke-width="0.5"/>
  <rect x="240" y="113" width="110" height="8" rx="1" fill="#14261f" stroke="#2d6b55" stroke-width="0.3"/>
  <rect x="240" y="124" width="110" height="8" rx="1" fill="#14261f" stroke="#2d6b55" stroke-width="0.3"/>
  <rect x="240" y="135" width="110" height="8" rx="1" fill="#14261f" stroke="#2d6b55" stroke-width="0.3"/>
  <rect x="240" y="146" width="110" height="8" rx="1" fill="#14261f" stroke="#2d6b55" stroke-width="0.3"/>
  <text x="295" y="174" fill="#8b98a5" font-size="7.5" text-anchor="middle">lookup table [max_seq, d]</text>
  <text x="295" y="190" fill="#8b98a5" font-size="7.5" text-anchor="middle">trained with model</text>
  <text x="295" y="206" fill="#8b98a5" font-size="7.5" text-anchor="middle">absolute position</text>
  <text x="295" y="226" fill="#6b7684" font-size="7" text-anchor="middle">GPT-2, BERT</text>
  <text x="295" y="242" fill="#6b7684" font-size="7" text-anchor="middle">simple, effective</text>
  <text x="295" y="258" fill="#f87171" font-size="7" text-anchor="middle">hard max_seq_len limit</text>
  <text x="295" y="270" fill="#f87171" font-size="7" text-anchor="middle">no extrapolation</text>

  <!-- ALiBi -->
  <rect x="390" y="70" width="170" height="210" rx="6" fill="#080d14" stroke="#f59e0b" stroke-width="1"/>
  <text x="475" y="90" fill="#fbbf24" font-size="9" font-weight="600" text-anchor="middle">ALiBi (2022)</text>
  <text x="475" y="112" fill="#8b98a5" font-size="7.5" text-anchor="middle">bias = -m · |i - j|</text>
  <text x="475" y="130" fill="#8b98a5" font-size="7.5" text-anchor="middle">linear penalty for distance</text>
  <!-- Distance penalty visualization -->
  <rect x="415" y="140" width="120" height="30" rx="2" fill="#0b1220" stroke="#334155" stroke-width="0.4"/>
  <text x="425" y="155" fill="#fbbf24" font-size="6.5">0</text>
  <text x="440" y="155" fill="#f59e0b" font-size="6.5">-m</text>
  <text x="460" y="155" fill="#f59e0b" font-size="6.5">-2m</text>
  <text x="483" y="155" fill="#b45309" font-size="6.5">-3m</text>
  <text x="508" y="155" fill="#92400e" font-size="6.5">-4m</text>
  <text x="475" y="190" fill="#8b98a5" font-size="7.5" text-anchor="middle">no parameters needed</text>
  <text x="475" y="206" fill="#8b98a5" font-size="7.5" text-anchor="middle">relative (implicit)</text>
  <text x="475" y="226" fill="#6b7684" font-size="7" text-anchor="middle">added to attn scores</text>
  <text x="475" y="242" fill="#6b7684" font-size="7" text-anchor="middle">MPT, BLOOM</text>
  <text x="475" y="258" fill="#34d399" font-size="7" text-anchor="middle">some extrapolation ability</text>

  <!-- RoPE -->
  <rect x="570" y="70" width="160" height="210" rx="6" fill="#080d14" stroke="#a78bfa" stroke-width="1.2"/>
  <text x="650" y="90" fill="#c4b5fd" font-size="9" font-weight="600" text-anchor="middle">RoPE (2021) ★</text>
  <!-- Rotation -->
  <circle cx="650" cy="135" r="25" fill="none" stroke="#a78bfa" stroke-width="0.8"/>
  <line x1="650" y1="135" x2="670" y2="118" stroke="#a78bfa" stroke-width="1.5"/>
  <circle cx="670" cy="118" r="3" fill="#a78bfa"/>
  <text x="650" y="174" fill="#8b98a5" font-size="7.5" text-anchor="middle">rotate Q,K by position angle</text>
  <text x="650" y="190" fill="#8b98a5" font-size="7.5" text-anchor="middle">relative via inner product</text>
  <text x="650" y="206" fill="#8b98a5" font-size="7.5" text-anchor="middle">multi-frequency encoding</text>
  <text x="650" y="226" fill="#6b7684" font-size="7" text-anchor="middle">Llama, Mistral, Qwen, all</text>
  <text x="650" y="242" fill="#34d399" font-size="7" text-anchor="middle">best extrapolation (YaRN)</text>
  <text x="650" y="258" fill="#34d399" font-size="7" text-anchor="middle">2024 consensus standard</text>

  <!-- Bottom comparison -->
  <rect x="30" y="292" width="700" height="138" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="380" y="312" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">Comparison Summary</text>
  <text x="120" y="335" fill="#8b98a5" font-size="8" font-weight="600" text-anchor="middle">Method</text>
  <text x="260" y="335" fill="#8b98a5" font-size="8" font-weight="600" text-anchor="middle">Type</text>
  <text x="400" y="335" fill="#8b98a5" font-size="8" font-weight="600" text-anchor="middle">Extrapolation</text>
  <text x="540" y="335" fill="#8b98a5" font-size="8" font-weight="600" text-anchor="middle">Params</text>
  <text x="670" y="335" fill="#8b98a5" font-size="8" font-weight="600" text-anchor="middle">Used by</text>
  <line x1="50" y1="342" x2="710" y2="342" stroke="#233043" stroke-width="0.5"/>
  <text x="120" y="358" fill="#60a5fa" font-size="8" text-anchor="middle">Sinusoidal</text>
  <text x="260" y="358" fill="#6b7684" font-size="8" text-anchor="middle">absolute, additive</text>
  <text x="400" y="358" fill="#f87171" font-size="8" text-anchor="middle">none</text>
  <text x="540" y="358" fill="#6b7684" font-size="8" text-anchor="middle">0</text>
  <text x="670" y="358" fill="#6b7684" font-size="8" text-anchor="middle">original Transformer</text>
  <text x="120" y="376" fill="#34d399" font-size="8" text-anchor="middle">Learned</text>
  <text x="260" y="376" fill="#6b7684" font-size="8" text-anchor="middle">absolute, additive</text>
  <text x="400" y="376" fill="#f87171" font-size="8" text-anchor="middle">none</text>
  <text x="540" y="376" fill="#6b7684" font-size="8" text-anchor="middle">max_seq×d</text>
  <text x="670" y="376" fill="#6b7684" font-size="8" text-anchor="middle">GPT-2, BERT</text>
  <text x="120" y="394" fill="#f59e0b" font-size="8" text-anchor="middle">ALiBi</text>
  <text x="260" y="394" fill="#6b7684" font-size="8" text-anchor="middle">relative, bias</text>
  <text x="400" y="394" fill="#f59e0b" font-size="8" text-anchor="middle">moderate</text>
  <text x="540" y="394" fill="#6b7684" font-size="8" text-anchor="middle">0</text>
  <text x="670" y="394" fill="#6b7684" font-size="8" text-anchor="middle">MPT, BLOOM</text>
  <text x="120" y="412" fill="#a78bfa" font-size="8" text-anchor="middle">RoPE ★</text>
  <text x="260" y="412" fill="#6b7684" font-size="8" text-anchor="middle">relative, multiplicative</text>
  <text x="400" y="412" fill="#34d399" font-size="8" text-anchor="middle">excellent (YaRN)</text>
  <text x="540" y="412" fill="#6b7684" font-size="8" text-anchor="middle">0</text>
  <text x="670" y="412" fill="#34d399" font-size="8" text-anchor="middle">all modern LLMs</text>

  <text x="380" y="452" fill="#6b7684" font-size="11" text-anchor="middle">Positional encoding is the bridge between set-based attention and sequence-aware language modeling.</text>
</svg>

Relative Positional Encodings:

• Relative Position Bias (T5, Transformer-XL): adds learned bias to attention logits based on relative distance between query and key; bias depends only on (i-j) not absolute positions i,j; typically uses bucketed distances (nearby positions get unique biases, distant positions share biases); generalizes better to longer sequences
• ALiBi (Attention with Linear Biases): adds constant bias -m·|i-j| to attention scores where m is head-specific slope; no learned parameters; extremely simple yet enables strong extrapolation; Llama 2 and many recent models use ALiBi; inference on 10× longer sequences than training with minimal degradation
• Relative Position Representations (Shaw et al.): adds learnable relative position embeddings to keys and values; attention(q_i, k_j) includes terms for both content and relative position; more expressive than bias-only methods but adds parameters
• DeBERTa Disentangled Attention: separates content and position attention; computes content-to-content, content-to-position, and position-to-content attention separately then combines; achieves state-of-the-art on many NLU benchmarks

Rotary Position Embedding (RoPE):

• Mechanism: rotates query and key vectors by angle proportional to position; for position m, rotate dimensions (2i, 2i+1) by angle m·θ_i where θ_i = 10000^(-2i/d); attention score naturally encodes relative position through dot product of rotated vectors
• Relative Position Property: dot product q_m^T k_n after rotation depends only on (m-n), providing relative position information without explicit bias terms; mathematically elegant and empirically effective
• Extrapolation: RoPE enables better length extrapolation than absolute encodings; with base frequency adjustment (increasing 10000 to larger values), models can extend to 8-32× training length; used in Llama, PaLM, GPT-NeoX, and most modern LLMs
• 2D/3D Extensions: RoPE generalizes to multi-dimensional positions; for images, apply separate rotations for height and width dimensions; for video, add temporal dimension; enables position-aware vision and video transformers

Advanced Position Encoding Techniques:

• xPos (Extrapolatable Position Encoding): modifies RoPE to include exponential decay based on relative distance; improves extrapolation by down-weighting very distant tokens; enables 10-20× length extrapolation with minimal perplexity increase
• Kerple (Kernelized Relative Position Encoding): uses kernel functions to compute position-dependent attention weights; combines benefits of relative position bias and RoPE; flexible framework encompassing many position encoding methods
• NoPE (No Position Encoding): some recent work shows that sufficiently large models can learn positional information from data alone without explicit encoding; requires careful attention to training data ordering and augmentation; controversial and not widely adopted
• Conditional Position Encoding: generates position encodings dynamically based on input content; enables position-aware processing that adapts to input structure (e.g., different encoding for code vs natural language)

Position Encoding for Different Modalities:

• Vision Transformers: 2D sinusoidal or learned position embeddings for patch positions; some models (DeiT) find that position encoding is less critical for vision than language; relative position bias (Swin) or no position encoding (ViT with sufficient data) can work well
• Audio/Speech: 1D position encoding similar to language; temporal position is critical for speech recognition and audio generation; some models use learnable convolutional position encoding that captures local temporal structure
• Graphs: position encoding for graph-structured data uses graph Laplacian eigenvectors, random walk statistics, or learned node embeddings; captures graph topology rather than sequential position
• Multimodal: different position encoding schemes for different modalities (2D for images, 1D for text); cross-modal attention must handle position encoding mismatch; some models use modality-specific position encodings that project to shared space

Practical Considerations:

• Training Efficiency: sinusoidal and ALiBi require no learned parameters, reducing memory and enabling immediate use at any sequence length; learned embeddings require storage and limit maximum length
• Inference Flexibility: RoPE and ALiBi enable efficient extrapolation to longer contexts; absolute learned embeddings require interpolation or extrapolation hacks that degrade quality
• Implementation Complexity: ALiBi is simplest (single line of code); RoPE requires careful implementation of rotation matrices; relative position bias requires managing bias tensors and bucketing logic

Positional encoding methods are a critical but often underappreciated component of Transformer architectures — the choice between absolute, relative, and rotary encodings fundamentally affects a model's ability to generalize to longer sequences, with modern approaches like RoPE and ALiBi enabling the multi-million token contexts that define frontier language models.