alibi positional encoding
**ALiBi (Attention with Linear Biases)** is the **positional encoding method that adds a static, non-learned linear penalty to attention scores based on the distance between query and key tokens**, replacing learned or sinusoidal position embeddings with a simple bias: attention_score(i,j) = q_i · k_j - m · |i - j|, where m is a head-specific slope that requires no training.
**Core Mechanism**: After computing raw attention scores Q·K^T, ALiBi subtracts a distance-proportional penalty:
score(i,j) = q_i · k_j - m_h · |i - j|
where m_h is a fixed slope for head h, set geometrically: m_h = 2^(-8h/H) for head h in {1,...,H}. Different heads attend to different distance scales: heads with small m values (large slopes) focus on recent tokens, heads with large m values (small slopes) attend broadly.
**Design Philosophy**: ALiBi argues that position information in transformers primarily serves to create a locality bias — recent tokens should be more relevant than distant ones. Rather than encoding absolute position into embeddings (which the model must learn to extract), ALiBi directly applies the desired recency bias as an attention score penalty.
**Comparison with Other Approaches**:
| Method | Mechanism | Parameters | Extrapolation | Overhead |
|--------|----------|-----------|--------------|----------|
| Sinusoidal | Add to embeddings | 0 | Poor | None |
| Learned absolute | Add to embeddings | N×d | None | Memory |
| RoPE | Rotate Q,K by position | 0 | Moderate | Compute |
| **ALiBi** | Subtract linear bias from scores | 0 | Strong | Minimal |
| T5 relative bias | Learned bias per distance | Buckets | Limited | Memory |
**Length Extrapolation**: ALiBi's strongest advantage. Because the linear penalty is defined for any distance, models trained with ALiBi can naturally extrapolate to longer sequences than seen during training. Empirical results show ALiBi models trained on 1024 tokens can evaluate on 2048+ tokens with minimal perplexity degradation — unlike sinusoidal or learned embeddings which degrade rapidly beyond training length.
**Per-Head Slopes**: The geometric progression of slopes (powers of 2^(-8/H)) creates a multi-scale attention pattern: low-slope heads have nearly uniform attention (global context), high-slope heads have sharply peaked attention (local context). This mirrors the observation that different attention heads in trained transformers naturally develop different locality patterns — ALiBi provides this inductive bias from initialization.
**Implementation Simplicity**: ALiBi requires no additional parameters, no special initialization, and no modification to the model architecture beyond adding a constant bias matrix to attention scores. The bias matrix can be precomputed once and cached. It integrates seamlessly with Flash Attention (the bias is applied within the tiling loop).
**Limitations**: ALiBi's linear distance penalty is a strong inductive bias that may be suboptimal for tasks requiring fine-grained position discrimination (e.g., counting, positional reasoning). RoPE provides richer position information through rotation, which may explain why most modern LLMs (LLaMA, Mistral) chose RoPE over ALiBi. ALiBi also makes attention strictly decrease with distance, which may not always be desirable (some tasks benefit from attending to specific distant positions).
**ALiBi demonstrated that positional encoding can be radically simplified to a parameter-free linear bias — its success challenged assumptions about what positional information transformers actually need, and its extrapolation properties influenced the development of more sophisticated length extension techniques for RoPE-based models.**