performer - kernel-based attention
# Performer - Kernel-Based Attention
## Introduction & Motivation
Performer: use kernel approximations for attention. Avoid explicit attention matrix computation. Applications: efficient transformers, long sequences.
Motivation: Reduce memory and time complexity of attention.
Applications: Long document processing, streaming inference.
---
## Core Concepts & Theory
### Kernel Approximation
Use kernel functions instead of softmax.
### Feature Maps
Approximate attention via kernel features.
### Orthogonal Features
Reduce approximation variance.
### Random Features
Compute efficiently without quadratic matrix.
---
## Mathematical Formulation
Softmax Attention:
$$ ext{Attn} = ext{softmax}(\frac{QK^T}{\sqrt{d}})V = \frac{e^{Q}(e^K)^T}{\mathbf{1}^T e^K}V$$
Kernel Approximation:
$$e^{x} \approx \phi(x) \cdot \phi(y)^T$$
Performer:
$$ ext{Attn} \approx \frac{\phi(Q)(\phi(K))^T V}{\phi(Q)(\phi(K))^T \mathbf{1}}$$
---
## Advanced Theory & Extensions
### Orthogonal Random Features
Reduced approximation error.
### Positive Features
Ensure positivity constraint.
### Generalized Kernels
Beyond softmax approximation.
---
## Computational Considerations
Standard attention: O(n²).
Performer: O(n·d·m) where m is feature dimension.
Space: O(n·m) instead of O(n²).
---
## Practical Implementation Strategies
### Feature Dimension Selection
Typical m = 256-512.
### Kernel Type
Random Fourier features typical.
### Warm-up Training
Careful initialization for stability.
---
## Benchmark Datasets & Evaluation
Long Range Arena: Long sequence benchmark.
Language Modeling: Perplexity metrics.
Inference Speed: Latency benchmarks.
---
## Key Challenges & Limitations
### Approximation Quality
May lose precision.
### Feature Dimension
Need sufficient features for accuracy.
### Numerical Stability
Carefully maintain stability.
---
## Hyperparameter Tuning
Feature dimension: 256-1024.
Kernel type: Random Fourier, orthogonal.
Learning rate: 1e-4 to 1e-3.
---
## Real-World Applications & Case Studies
Long Documents: Process 4096+ length sequences.
Streaming: Process online without storing history.
Large Models: Reduce memory footprint.
---
## Integration with Other Methods
Performer + sparse attention; + hierarchical features.
---
## Summary & Key Takeaways
Performer achieves efficient attention via kernels.
Principles:
1. Kernel approximation: Feature maps.
2. Linear complexity: O(n) time and space.
3. Random features: Efficient computation.
4. Approximation: Maintain expressiveness.
5. Practical: Long sequence processing.
---
## Appendix: Practical Labs
### Lab 1: Kernel Attention
import numpy as np
def kernel_attention(query, key, value, num_features=256):
"""Compute attention via kernel approximation"""
# Generate random features
D = query.shape[-1]
w = np.random.randn(num_features, D)
b = np.random.uniform(0, 2*np.pi, num_features)
# Feature maps: cos((wx+b))
q_feat = np.cos(query @ w.T + b)
k_feat = np.cos(key @ w.T + b)
# Approximate attention
numerator = q_feat @ (k_feat.T @ value)
denominator = q_feat @ k_feat.T.sum(axis=1, keepdims=True)
output = numerator / (denominator + 1e-8)
return output
np.random.seed(42)
q = np.random.randn(100, 64)
k = np.random.randn(100, 64)
v = np.random.randn(100, 64)
out = kernel_attention(q, k, v)
assert out.shape == (100, 64)
print("✓ Kernel attention working")### Lab 2: Random Fourier Features
import numpy as np
def random_fourier_features(x, num_features=256):
"""Compute random Fourier features"""
D = x.shape[-1]
# Random projection matrix
w = np.random.randn(num_features, D) / np.sqrt(D)
b = np.random.uniform(0, 2*np.pi, num_features)
# RFF: sqrt(2/m) * cos(Wx + b)
features = np.sqrt(2 / num_features) * np.cos(x @ w.T + b)
return features
np.random.seed(42)
x = np.random.randn(100, 64)
feats = random_fourier_features(x, 256)
assert feats.shape == (100, 256)
print("✓ Random Fourier features working")### Lab 3: Orthogonal Features
import numpy as np
def orthogonal_random_features(x, num_features=256):
"""Compute orthogonal random features"""
D = x.shape[-1]
# Orthogonal matrix via QR
w_full = np.random.randn(num_features, D)
w, _ = np.linalg.qr(w_full)
w = w[:D, :].T # Take first D rows and transpose
b = np.random.uniform(0, 2*np.pi, D)
# Features
features = np.cos(x @ w + b)
return features
np.random.seed(42)
x = np.random.randn(100, 64)
feats = orthogonal_random_features(x, 64)
assert feats.shape[1] == 64
print("✓ Orthogonal features working")### Lab 4: Approximation Error
import numpy as np
def estimate_approximation_error(query, key, value, num_features=256):
"""Estimate approximation error vs standard attention"""
# Standard attention
d_k = key.shape[-1]
scores = query @ key.T / np.sqrt(d_k)
attn_standard = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
output_standard = attn_standard @ value
# Kernel attention (simplified)
q_feat = query @ np.random.randn(query.shape[-1], num_features)
k_feat = key @ np.random.randn(key.shape[-1], num_features)
attn_kernel = q_feat @ k_feat.T / (q_feat.sum() * k_feat.sum() + 1e-8)
output_kernel = attn_kernel @ value
# MSE error
error = np.mean((output_standard - output_kernel) ** 2)
return error
np.random.seed(42)
q = np.random.randn(50, 64)
k = np.random.randn(50, 64)
v = np.random.randn(50, 64)
err = estimate_approximation_error(q, k, v)
assert err >= 0
print(f"✓ Approximation error: {err:.4f}")---