Performer is the kernelized attention mechanism that rewrites softmax into feature maps so Vision Transformers get linear-time attention without bias — it approximates the exponential kernel with FAVOR+ random features, enabling the attention to be computed as (phi(Q) (phi(K)^T V)) instead of explicitly building the full similarity matrix.

What Is Performer?

• Definition: A transformer block that maps queries, keys, and values to kernel feature spaces using random projections or orthogonal features, then computes attention via associative matrix multiplications.
• Key Feature 1: Random Fourier features guarantee positive and unbiased estimates for softmax kernels.
• Key Feature 2: Attention becomes associative so that contexts can be accumulated incrementally, enabling streaming inference and very long sequences.
• Key Feature 3: Extra normalization steps (like causal masks or epsilon smoothing) keep the approximation stable.
• Key Feature 4: Predictor kernels can be parameterized as ReLU features or generalized linear transformations tuned during training.

Why Performer Matters

• Linear Memory and Compute: Complexity shrinks to O(Nd), so gigapixel images or lengthy video clips no longer tax GPU memory walls.
• Streaming and Sampling: Because attention can be accumulated in chunks, Performer suits autoregressive decoding with unbounded context windows.
• Bias-Free Approximation: Unlike some sparse attention patterns, the kernel estimate remains unbiased, so gradients focus on the right dependencies.
• Generalization: Empirical studies show that Performer matches softmax attention on language and vision tasks with only modestly more features.
• Hardware Efficiency: Matmul-heavy computations remain friendly to tensor cores without the need to materialize large softmax matrices.

Kernel Choices

Positive Random Features:

• Use φ(x) = elu(x) + 1 or exp(x) approximations with random Gaussian projections.
• Guarantee positive outputs so the attention remains well-defined.

Orthogonal Features:

• Apply QR decomposition to random projection matrices for lower variance.
• Spread randomness evenly across feature dimensions.

Deterministic Features:

• Instead of random draws, use structured matrices (like Rader transforms) for reproducible kernels.

How It Works / Technical Details

Step 1: Project queries and keys through the kernel map φ, producing positive vectors of dimension m; compute the numerator by multiplying φ(Q)^T with V and the denominator by summing φ(K) across tokens.

Step 2: For causal settings, apply prefix sums so that each token only attends to previous ones. Then divide the numerator by the denominator and continue with the usual feed-forward and normalization layers.

Comparison / Alternatives

Tools & Platforms

• Performer-PyTorch: Reference implementation with FAVOR+ kernels for vision tasks.
• DeepSpeed: Integrates Performer blocks inside ZeRO pipelines for efficient training.
• TensorFlow Addons: Contains kernel functions for positive random features.
• Fairseq / Hugging Face: Provide configs and weights to swap out standard attention.

Performer is the kernel trick that lets transformers see without quadratic baggage — it rewrites self-attention into a sequence of matmuls that never expand the N×N matrix even when N reaches tens of thousands.