ALiBi (Attention with Linear Biases) is a positional encoding method for Transformers that replaces learned or sinusoidal positional embeddings with a simple linear penalty added directly to attention scores, where the penalty is proportional to the distance between the query and key tokens. ALiBi adds a bias of -m·|i-j| to the attention logit between positions i and j, where m is a fixed, head-specific slope that varies geometrically across attention heads.

Why ALiBi Matters in AI/ML: ALiBi enables superior length extrapolation compared to other positional encodings, allowing models trained on short sequences to generalize to much longer sequences at inference time with minimal performance degradation, addressing a critical limitation of standard positional encodings.

• Linear distance penalty — The attention score becomes softmax(q_i^T·k_j - m·|i-j|), where the linear bias penalizes attending to distant tokens; this implements a soft local attention window whose effective width varies across heads due to different slope values m • Head-specific slopes — Slopes are set to geometric sequence m_h = 1/2^(h·8/H) for H heads (e.g., for 8 heads: 1/2, 1/4, 1/8, ..., 1/256); heads with large slopes focus on nearby tokens (local patterns), while heads with small slopes attend to distant tokens (global patterns) • Zero additional parameters — ALiBi requires no learned parameters for position encoding: slopes are fixed constants, and no positional embeddings are added to input tokens; this simplifies the model and reduces memory usage • Length extrapolation — Models trained with ALiBi on sequences of length L can effectively process sequences of 2-4× L at inference time with graceful degradation, because the linear bias provides a smooth inductive bias for unseen distances rather than undefined embeddings • No position embeddings — Unlike sinusoidal, learned, or RoPE encodings that modify token representations, ALiBi operates entirely in the attention logit space; input tokens are position-agnostic, and all positional information is injected at the attention computation

ALiBi is the simplest and most effective method for achieving length extrapolation in Transformers, replacing complex positional embeddings with a parameter-free linear attention bias that provides multi-scale distance awareness across heads and enables models to generalize to sequence lengths far beyond their training context.