performer for vision
**Performer** is the **kernelized attention mechanism that rewrites softmax into feature maps so Vision Transformers get linear-time attention without bias** — it approximates the exponential kernel with FAVOR+ random features, enabling the attention to be computed as (phi(Q) (phi(K)^T V)) instead of explicitly building the full similarity matrix.
**What Is Performer?**
- **Definition**: A transformer block that maps queries, keys, and values to kernel feature spaces using random projections or orthogonal features, then computes attention via associative matrix multiplications.
- **Key Feature 1**: Random Fourier features guarantee positive and unbiased estimates for softmax kernels.
- **Key Feature 2**: Attention becomes associative so that contexts can be accumulated incrementally, enabling streaming inference and very long sequences.
- **Key Feature 3**: Extra normalization steps (like causal masks or epsilon smoothing) keep the approximation stable.
- **Key Feature 4**: Predictor kernels can be parameterized as ReLU features or generalized linear transformations tuned during training.
**Why Performer Matters**
- **Linear Memory and Compute**: Complexity shrinks to O(Nd), so gigapixel images or lengthy video clips no longer tax GPU memory walls.
- **Streaming and Sampling**: Because attention can be accumulated in chunks, Performer suits autoregressive decoding with unbounded context windows.
- **Bias-Free Approximation**: Unlike some sparse attention patterns, the kernel estimate remains unbiased, so gradients focus on the right dependencies.
- **Generalization**: Empirical studies show that Performer matches softmax attention on language and vision tasks with only modestly more features.
- **Hardware Efficiency**: Matmul-heavy computations remain friendly to tensor cores without the need to materialize large softmax matrices.
**Kernel Choices**
**Positive Random Features**:
- Use φ(x) = elu(x) + 1 or exp(x) approximations with random Gaussian projections.
- Guarantee positive outputs so the attention remains well-defined.
**Orthogonal Features**:
- Apply QR decomposition to random projection matrices for lower variance.
- Spread randomness evenly across feature dimensions.
**Deterministic Features**:
- Instead of random draws, use structured matrices (like Rader transforms) for reproducible kernels.
**How It Works / Technical Details**
**Step 1**: Project queries and keys through the kernel map φ, producing positive vectors of dimension m; compute the numerator by multiplying φ(Q)^T with V and the denominator by summing φ(K) across tokens.
**Step 2**: For causal settings, apply prefix sums so that each token only attends to previous ones. Then divide the numerator by the denominator and continue with the usual feed-forward and normalization layers.
**Comparison / Alternatives**
| Aspect | Performer | Linformer | Windowed / Axial |
|--------|-----------|-----------|-----------------|
| Complexity | O(N d) | O(N k) | O(N w^2) or O(N(H+W)) |
| Approximation Bias | Zero | Low similar patterns | None but no compression |
| Suitability | Streaming + long context | Low-rank scenes | Structured spatial data |
| Hardware | Matmul-friendly | Matmul-friendly | Requires extra reshapes |
**Tools & Platforms**
- **Performer-PyTorch**: Reference implementation with FAVOR+ kernels for vision tasks.
- **DeepSpeed**: Integrates Performer blocks inside ZeRO pipelines for efficient training.
- **TensorFlow Addons**: Contains kernel functions for positive random features.
- **Fairseq / Hugging Face**: Provide configs and weights to swap out standard attention.
Performer is **the kernel trick that lets transformers see without quadratic baggage** — it rewrites self-attention into a sequence of matmuls that never expand the N×N matrix even when N reaches tens of thousands.