positional encoding

Positional encoding informs models about token positions in sequences enabling attention mechanisms to use order information. Absolute positional encoding adds position-specific vectors to token embeddings. Learned positional embeddings are trained parameters. Sinusoidal encoding uses sine and cosine functions at different frequencies. Relative positional encoding represents distances between tokens rather than absolute positions. RoPE Rotary Position Embedding rotates token embeddings based on position enabling length extrapolation beyond training context. ALiBi Attention with Linear Biases adds position-dependent bias to attention scores. These methods enable models to generalize to longer sequences than seen during training. RoPE is used in Llama and many modern models. ALiBi is used in BLOOM. Positional encoding is critical for transformers which otherwise treat sequences as sets. Without it models cannot distinguish token order. Length extrapolation is important for long-context applications. RoPE and ALiBi enable models trained on 2K contexts to handle 32K or more. Positional encoding design significantly impacts model capabilities especially for long sequences.