linear attention for vision
**Linear Attention** is the **kernel-based trick that rewrites attention as a sequence of associative matrix products so complexity grows linearly with token count** — it replaces the softmax with a positive-definite kernel, enabling ViTs to process extremely long sequences without quadratic memory while still capturing contextual dependencies.
**What Is Linear Attention?**
- **Definition**: A reformulation of attention where queries and keys are passed through feature maps φ, and attention is computed as φ(Q) (φ(K)^T V), eliminating the explicit N×N similarity matrix.
- **Key Feature 1**: The kernel map φ is chosen so that the resultant context computation is associative, allowing context accumulation in streaming models.
- **Key Feature 2**: Token ordering is preserved through positional encodings added before kernel projection.
- **Key Feature 3**: The method remains unbiased if φ produces positive outputs that mimic softmax weights.
- **Key Feature 4**: Linear attention handles varying sequence lengths with constant additional memory.
**Why Linear Attention Matters**
- **Memory Efficiency**: Removes the need for O(N^2) storage, making high-resolution vision and video viable.
- **Speed**: Faster on long sequences since it reduces the number of similarity computations.
- **Streaming Friendly**: Context can be updated incrementally because the operations are associative.
- **Bias-Free**: Unlike sparse attention, linear attention does not drop any tokens and keeps all terms.
- **Compatibility**: Integrates well with causal decoding and is easy to implement in modern frameworks.
**Kernel Choices**
**Positive Random Features**:
- φ(x) = elu(x) + 1 or softplus approximations to keep outputs positive.
- Works for both language and vision when scaled appropriately.
**Quadratic Polynomial Features**:
- Use polynomial kernels for structured interactions.
- Provide deterministic approximations with low variance.
**Learnable Kernels**:
- Parameterize the kernel map and train it end to end.
- Allows the network to discover the best feature projections.
**How It Works / Technical Details**
**Step 1**: Transform queries and keys through φ to obtain features of dimension m, then compute the context numerator as (φ(K)^T V) and denominator as sum(φ(K)) per position.
**Step 2**: Multiply φ(Q) with the numerator, divide by the denominator (ensuring positivity), and apply softmax-like normalization before projecting back to the model space.
**Comparison / Alternatives**
| Aspect | Linear Attention | Softmax Attention | Sparse Attention |
|--------|------------------|-------------------|------------------|
| Complexity | O(N) | O(N^2) | O(Nk)
| Bias | None if kernel proper | None | Approximation
| Long Sequence | Excellent | Poor | Limited
| Implementation | Slightly complex | Standard | Moderate
**Tools & Platforms**
- **Performer**: Uses FAVOR+ kernels to implement linear attention.
- **Linear Transformer libraries**: Provide kernel maps ready for ViT blocks.
- **inference engines**: TensorRT kernels now include linear attention for transformers.
- **Debugging**: Monitor denominator values to prevent division by zero when kernels produce small sums.
Linear attention is **the linear pathway that lets transformers remain faithful to every token without blowing up compute for massive images** — it achieves the same context mixing as softmax but with a tame resource profile.