Positional Encoding for Transformers

Why Positional Encoding? Transformers have no inherent notion of sequence order. Positional encoding injects position information so the model knows where each token is in the sequence.

Encoding Methods

Sinusoidal Positional Encoding (Original Transformer) $$ PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d}) $$ $$ PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d}) $$

• Fixed, not learned
• Can extrapolate to longer sequences (in theory)
• Added to token embeddings

Learned Positional Embeddings

• Trainable embedding for each position
• Used in GPT-2, BERT
• Cannot extrapolate beyond training length

RoPE (Rotary Position Embedding) Used by: Llama, Mistral, Qwen, and most modern models

Key ideas:

• Encodes position in the rotation of query and key vectors
• Relative position naturally emerges from the dot product
• Better length extrapolation than absolute encodings

# Simplified RoPE application
def apply_rope(x, freqs):
    # Split into pairs, rotate by position-dependent angle
    x_rotated = rotate_half(x) * freqs
    return x * torch.cos(freqs) + x_rotated * torch.sin(freqs)

ALiBi (Attention with Linear Biases) Used by: MPT, BLOOM

• No position encoding in embeddings
• Subtracts linear bias from attention scores based on distance
• Excellent extrapolation properties
• Simple: $score_{ij} = q_i \cdot k_j - m \cdot |i - j|$

Comparison

Length Extrapolation RoPE can be extended with:

• Linear scaling: Divide positions by factor
• NTK-aware scaling: Adjust frequency base
• YaRN: Position interpolation with attention scaling