random feature attention
**Random Feature Attention** is an approach to efficient attention that replaces the explicit computation of the N×N attention matrix with random feature map approximations of the softmax kernel, enabling linear-time attention by decomposing the exponential kernel into a dot product of random projections. This encompasses methods like Performer's FAVOR+, Random Feature Attention (RFA), and related kernel approximation techniques that share the mathematical framework of representing softmax as an inner product in a randomized feature space.
**Why Random Feature Attention Matters in AI/ML:**
Random feature attention provides a **mathematically grounded approach to linear attention** that maintains the non-negativity and normalization properties of softmax while reducing quadratic complexity, offering provable approximation bounds.
• **Random Fourier Features (RFF)** — Bochner's theorem guarantees that any shift-invariant kernel k(x-y) can be approximated as φ(x)^T φ(y) using φ(x) = √(2/m)·[cos(ω₁^T x + b₁), ..., cos(ω_m^T x + b_m)] with ω_i sampled from the kernel's spectral density
• **Positive random features** — For softmax attention (which requires non-negative weights), positive random features φ(x) = exp(ωᵢ^T x - ||x||²/2)/√m ensure all attention weights are positive, preserving the probability distribution interpretation of attention
• **Approximation quality vs. features** — The kernel approximation error scales as O(1/√m) for m random features; m=256 typically achieves <5% relative error on the attention matrix for d=64 head dimensions
• **Gated attention variants** — Some methods combine random feature attention with gating mechanisms that control information flow, compensating for approximation errors in the attention weights with learned gates
• **Causal masking with prefix sums** — Random feature attention supports causal (autoregressive) masking through cumulative sum operations: S_t = Σ_{s≤t} φ(k_s)·v_s^T and z_t = Σ_{s≤t} φ(k_s), enabling O(1) per-step generation
| Method | Feature Type | Non-Negative | Approximation Quality |
|--------|-------------|-------------|----------------------|
| RFF (Fourier) | cos(ω^T x + b) | No | Good (Gaussian kernel) |
| FAVOR+ (Performer) | exp(ω^T x) | Yes | Good (softmax) |
| RFA (gated) | Softmax RFF + gating | Yes | Very good |
| Positive RFF | exp(ω^T x - ||x||²/2) | Yes | Good |
| Deterministic features | Learned projections | Varies | Architecture-dependent |
| Hybrid (local + random) | RFF + local window | Yes | Excellent |
**Random feature attention provides the mathematical foundation for linearizing softmax attention through kernel approximation theory, enabling O(N) attention computation with provable error bounds that decrease with the number of random features, establishing the theoretical basis for efficient, scalable Transformer architectures.**