Reformer - Lsh Attention

# Reformer - LSH Attention

## Introduction & Motivation

Reformer: use locality-sensitive hashing for attention. Attend to similar positions. Applications: efficient transformers for long sequences.

Motivation: Reduce attention computation via locality hashing.

Applications: Long document processing, memory-efficient transformers.

---

## Core Concepts & Theory

### Locality-Sensitive Hashing

Hash similar items to same bucket.

### Bucketing

Group positions into buckets.

### Sorted Bucket Attention

Attend within and across buckets.

### Hash Function

Map positions to buckets efficiently.

---

## Mathematical Formulation

Hash Function:
$$h(x) = \lfloor \frac{x \cdot v}{b} floor$$

Bucket Assignment:
$$ ext{bucket}(i) = h(k_i)$$

Attention Computation:
$$ ext{Attention}_{ ext{bucket}} = ext{softmax}(\frac{QK^T_{ ext{bucket}}}{\sqrt{d}})V_{ ext{bucket}}$$

---

## Advanced Theory & Extensions

### Multi-Hash

Use multiple hash functions.

### Revolver Attention

Attend to adjacent positions.

### Chunking

Process sequences in chunks.

---

## Computational Considerations

Bucketing: O(n log n).

Attention per bucket: O(b²·d) where b = bucket size.

Total: O(n·b·d) instead of O(n²·d).

---

## Practical Implementation Strategies

### Bucket Size Selection

Typically 32-64 positions.

### Hash Function Choice

Random projection common.

### Overflow Handling

Handle hash collisions.

---

## Benchmark Datasets & Evaluation

Long Range Arena: Long sequence benchmark.

Language Modeling: Perplexity.

Speed Benchmarks: Inference efficiency.

---

## Key Challenges & Limitations

### Hash Collision

Similar items in same bucket.

### Bucket Balance

Uneven bucket sizes.

### Sequential Dependency

Fixed bucketing order.

---

## Hyperparameter Tuning

Number of hashes: 1-3.

Bucket size: 32-128.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Long Documents: 4096+ length.

Memory Efficiency: Reduced memory footprint.

Fast Training: Accelerated pre-training.

---

## Integration with Other Methods

Reformer + reversible layers; + chunking for efficiency.

---

## Summary & Key Takeaways

Reformer enables efficient attention via LSH.

Principles:
1. LSH: Hash similar positions.
2. Bucketing: Group into buckets.
3. Local attention: Attend within buckets.
4. Efficiency: O(n log n) computation.
5. Practical: Long sequence processing.

---

## Appendix: Practical Labs

### Lab 1: LSH Bucketing

import numpy as np

def lsh_hash(positions, hash_dim=64, num_buckets=32):
 """Hash positions to buckets using LSH"""
 # Random projection
 v = np.random.randn(hash_dim)
 v = v / np.linalg.norm(v)
 
 # Project and bucket
 projections = positions @ v
 buckets = np.floor(projections / (np.max(projections) / num_buckets)).astype(int)
 
 return buckets

np.random.seed(42)
pos = np.random.randn(100, 64)
buckets = lsh_hash(pos, 64, 32)
assert len(np.unique(buckets)) <= 32
print(f"✓ LSH bucketing: {len(np.unique(buckets))} unique buckets")

### Lab 2: Bucket Attention

import numpy as np

def attend_within_buckets(query, key, value, buckets):
 """Attend within bucketed positions"""
 unique_buckets = np.unique(buckets)
 outputs = []
 
 for bucket_id in unique_buckets:
 mask = buckets == bucket_id
 indices = np.where(mask)[0]
 
 if len(indices) > 0:
 q_bucket = query[indices]
 k_bucket = key[indices]
 v_bucket = value[indices]
 
 # Compute attention within bucket
 scores = q_bucket @ k_bucket.T / np.sqrt(query.shape[-1])
 attn = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
 output_bucket = attn @ v_bucket
 outputs.append((indices, output_bucket))
 
 return outputs

np.random.seed(42)
q = np.random.randn(100, 64)
k = np.random.randn(100, 64)
v = np.random.randn(100, 64)
buckets = np.random.randint(0, 32, 100)
outputs = attend_within_buckets(q, k, v, buckets)
assert len(outputs) > 0
print(f"✓ Bucket attention: {len(outputs)} buckets")

### Lab 3: Multi-Hash

import numpy as np

def multi_hash_lsh(positions, num_hashes=3, num_buckets=32):
 """Use multiple hash functions"""
 hash_results = []
 
 for h in range(num_hashes):
 v = np.random.randn(positions.shape[-1])
 v = v / np.linalg.norm(v)
 
 projections = positions @ v
 buckets = np.floor(projections / (np.max(projections) / num_buckets)).astype(int)
 hash_results.append(buckets)
 
 return hash_results

np.random.seed(42)
pos = np.random.randn(100, 64)
hashes = multi_hash_lsh(pos, num_hashes=3)
assert len(hashes) == 3
print("✓ Multi-hash LSH working")

### Lab 4: Bucket Statistics

import numpy as np

def analyze_bucket_distribution(buckets, num_buckets=32):
 """Analyze bucket size distribution"""
 bucket_sizes = []
 
 for b in range(num_buckets):
 size = np.sum(buckets == b)
 if size > 0:
 bucket_sizes.append(size)
 
 if len(bucket_sizes) > 0:
 mean_size = np.mean(bucket_sizes)
 max_size = np.max(bucket_sizes)
 min_size = np.min(bucket_sizes)
 return mean_size, max_size, min_size
 else:
 return 0, 0, 0

np.random.seed(42)
buckets = np.random.randint(0, 32, 1000)
mean, max_s, min_s = analyze_bucket_distribution(buckets)
print(f"✓ Bucket stats: mean={mean:.1f}, max={max_s}, min={min_s}")

---