Linear Attention is the kernel-based trick that rewrites attention as a sequence of associative matrix products so complexity grows linearly with token count — it replaces the softmax with a positive-definite kernel, enabling ViTs to process extremely long sequences without quadratic memory while still capturing contextual dependencies.

What Is Linear Attention?

• Definition: A reformulation of attention where queries and keys are passed through feature maps φ, and attention is computed as φ(Q) (φ(K)^T V), eliminating the explicit N×N similarity matrix.
• Key Feature 1: The kernel map φ is chosen so that the resultant context computation is associative, allowing context accumulation in streaming models.
• Key Feature 2: Token ordering is preserved through positional encodings added before kernel projection.
• Key Feature 3: The method remains unbiased if φ produces positive outputs that mimic softmax weights.
• Key Feature 4: Linear attention handles varying sequence lengths with constant additional memory.

Why Linear Attention Matters

• Memory Efficiency: Removes the need for O(N^2) storage, making high-resolution vision and video viable.
• Speed: Faster on long sequences since it reduces the number of similarity computations.
• Streaming Friendly: Context can be updated incrementally because the operations are associative.
• Bias-Free: Unlike sparse attention, linear attention does not drop any tokens and keeps all terms.
• Compatibility: Integrates well with causal decoding and is easy to implement in modern frameworks.

Kernel Choices

Positive Random Features:

• φ(x) = elu(x) + 1 or softplus approximations to keep outputs positive.
• Works for both language and vision when scaled appropriately.

Quadratic Polynomial Features:

• Use polynomial kernels for structured interactions.
• Provide deterministic approximations with low variance.

Learnable Kernels:

• Parameterize the kernel map and train it end to end.
• Allows the network to discover the best feature projections.

How It Works / Technical Details

Step 1: Transform queries and keys through φ to obtain features of dimension m, then compute the context numerator as (φ(K)^T V) and denominator as sum(φ(K)) per position.

Step 2: Multiply φ(Q) with the numerator, divide by the denominator (ensuring positivity), and apply softmax-like normalization before projecting back to the model space.

Comparison / Alternatives

Tools & Platforms

• Performer: Uses FAVOR+ kernels to implement linear attention.
• Linear Transformer libraries: Provide kernel maps ready for ViT blocks.
• inference engines: TensorRT kernels now include linear attention for transformers.
• Debugging: Monitor denominator values to prevent division by zero when kernels produce small sums.

Linear attention is the linear pathway that lets transformers remain faithful to every token without blowing up compute for massive images — it achieves the same context mixing as softmax but with a tame resource profile.