linear attention
**Linear Attention** is a family of attention mechanisms that approximate or replace the standard softmax attention with computations that scale linearly O(N) in sequence length rather than quadratically O(N²), enabling Transformers to process much longer sequences within practical memory and compute budgets. Linear attention achieves this by decomposing the attention operation so that queries, keys, and values can be combined without explicitly computing the full N×N attention matrix.
**Why Linear Attention Matters in AI/ML:**
Linear attention addresses the **fundamental scalability bottleneck** of Transformers—the quadratic cost of full attention—enabling efficient processing of long sequences (documents, high-resolution images, genomics) that are computationally prohibitive with standard attention.
• **Kernel trick decomposition** — Standard attention computes softmax(QK^T)V, requiring the N×N matrix QK^T; linear attention replaces softmax with a kernel: Attn(Q,K,V) = φ(Q)(φ(K)^T V), where φ(K)^T V can be computed first in O(N·d²) instead of O(N²·d)
• **Right-to-left association** — The key insight: by computing (K^T V) first (d×d matrix), then multiplying with Q, the computation avoids materializing the N×N attention matrix; this changes associativity from (QK^T)V to Q(K^T V), reducing complexity from O(N²d) to O(Nd²)
• **Feature map choice** — The kernel function φ(·) determines approximation quality; common choices include: elu(x)+1, random Fourier features (Performer), polynomial kernels, and learned feature maps; the choice affects expressiveness-efficiency tradeoff
• **Recurrent formulation** — Linear attention can be reformulated as a recurrent neural network: S_t = S_{t-1} + k_t v_t^T (state update), o_t = q_t^T S_t (output); this enables O(1) per-step inference for autoregressive generation
• **Quality-efficiency tradeoff** — Linear attention is faster but generally less expressive than softmax attention; softmax provides sparse, data-dependent attention patterns while linear attention produces smoother, more uniform patterns
| Method | Complexity | Feature Map | Quality vs Softmax |
|--------|-----------|-------------|-------------------|
| Standard Softmax | O(N²d) | exp(QK^T/√d) | Baseline |
| Linear (ELU+1) | O(Nd²) | elu(x) + 1 | Lower (smooth attention) |
| Performer (FAVOR+) | O(Nd) | Random Fourier features | Moderate |
| cosFormer | O(Nd²) | cos-weighted linear | Good |
| TransNormer | O(Nd²) | Normalization-based | Good |
| RetNet | O(Nd²) | Exponential decay | Strong |
**Linear attention is the key algorithmic innovation for scaling Transformers beyond quadratic complexity, replacing the N×N attention matrix with decomposed kernel computations that enable linear-time sequence processing while maintaining the core attention mechanism's ability to model token interactions across the sequence.**