Linformer - Linear Attention Complexity
# Linformer - Linear Attention Complexity
## Introduction & Motivation
Linformer: reduce attention complexity from quadratic to linear. Project key-value to lower dimensions. Applications: efficient transformers for long sequences.
Motivation: Enable transformers for long sequences efficiently.
Applications: Long document understanding, streaming processing.
---
## Core Concepts & Theory
### Linear Complexity
Reduce from O(n²) to O(n).
### Projection Trick
Project attention to lower rank.
### Approximate Attention
Maintain expressiveness with fewer computations.
### Efficiency Gain
4-9x speedup on long sequences.
---
## Mathematical Formulation
Standard Attention:
$$ ext{Attn} = ext{softmax}(\frac{QK^T}{\sqrt{d}})V, \quad O(n^2)$$
Linformer Projection:
$$K' = K \cdot P, V' = V \cdot Q ext{ where } P, Q \in \mathbb{R}^{n imes k}$$
Linear Attention:
$$ ext{Attn} = ext{softmax}(\frac{Q(K')^T}{\sqrt{d}})(V'), \quad O(nk)$$
---
## Advanced Theory & Extensions
### Dynamic Projection
Learn projection matrices.
### Multi-Head Projection
Separate projections per head.
### Hybrid Attention
Combine local and linear global.
---
## Computational Considerations
Standard: O(n²·d).
Linformer: O(n·k·d) where k << n.
Speedup: ~8× for n=4096, k=256.
---
## Practical Implementation Strategies
### Projection Dimension Selection
Typical k = 256-512.
### Layer-wise Projection
Different projections per layer.
### Warm-up Training
Careful initialization.
---
## Benchmark Datasets & Evaluation
Long Document: SCROLLS benchmark.
Language Modeling: Wikitext, Penn Treebank.
Speed Benchmarks: Inference latency.
---
## Key Challenges & Limitations
### Approximation Error
May lose long-range dependencies.
### Projection Quality
Fixed vs learned projections.
### Theoretical Justification
Approximation guarantees.
---
## Hyperparameter Tuning
Projection dimension: 256-512.
Projection type: random, learned.
Learning rate: 1e-4 to 1e-3.
---
## Real-World Applications & Case Studies
Long Documents: Process beyond context limit.
Streaming: Process long sequences online.
Large Batch: Handle bigger models.
---
## Integration with Other Methods
Linformer + sparse attention; + local attention for locality.
---
## Summary & Key Takeaways
Linformer achieves linear attention complexity.
Principles:
1. Projection: Reduce key-value dimension.
2. Linear complexity: O(nk) computation.
3. Approximation: Maintain expressiveness.
4. Efficiency: 8× speedup.
5. Practical: Works on long sequences.
---
## Appendix: Practical Labs
### Lab 1: Projection-Based Attention
import numpy as np
def linformer_attention(query, key, value, proj_dim=256):
"""Compute linear attention via projection"""
n, d = query.shape
# Project key and value
P = np.random.randn(n, proj_dim) / np.sqrt(proj_dim)
key_proj = key @ P
Q = np.random.randn(n, proj_dim) / np.sqrt(proj_dim)
value_proj = value @ Q
# Linear attention
d_k = query.shape[-1]
scores = query @ key_proj.T / np.sqrt(d_k)
attention = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
output = attention @ value_proj
return output
np.random.seed(42)
q = np.random.randn(100, 64)
k = np.random.randn(100, 64)
v = np.random.randn(100, 64)
out = linformer_attention(q, k, v, proj_dim=32)
assert out.shape == (100, 64)
print("✓ Linformer attention working")### Lab 2: Projection Dimension Selection
def optimal_projection_dim(seq_len, target_ratio=0.1):
"""Select projection dimension based on sequence length"""
proj_dim = max(int(seq_len * target_ratio), 32)
proj_dim = min(proj_dim, seq_len) # Cap at seq_len
return proj_dim
for seq_len in [100, 1000, 4096]:
proj_dim = optimal_projection_dim(seq_len)
assert proj_dim <= seq_len
print(f"Seq len {seq_len} → proj dim {proj_dim}")
print("✓ Projection dimension selection working")### Lab 3: Complexity Comparison
def compare_attention_complexity(n, d, proj_dim):
"""Compare standard vs linear attention complexity"""
standard_flops = n * n * d
linear_flops = n * proj_dim * d + n * d * proj_dim
speedup = standard_flops / linear_flops
return speedup
speedup_100 = compare_attention_complexity(100, 64, 32)
speedup_4096 = compare_attention_complexity(4096, 64, 256)
assert speedup_4096 > speedup_100 # Greater speedup for longer sequences
print(f"✓ Speedup: seq_len=4096: {speedup_4096:.1f}x")### Lab 4: Learned Projection
import numpy as np
def learned_projection(key, value, num_projections=4):
"""Learn multiple projections"""
projections = [np.random.randn(key.shape[0], 256) for _ in range(num_projections)]
# Average multiple projections
key_proj = np.mean([key @ p for p in projections], axis=0)
value_proj = np.mean([value @ p for p in projections], axis=0)
return key_proj, value_proj
np.random.seed(42)
k = np.random.randn(100, 64)
v = np.random.randn(100, 64)
k_p, v_p = learned_projection(k, v)
assert k_p.shape[0] == 100
print("✓ Learned projection working")---