content-based sparse attention

**Content-Based Sparse Attention** is a **dynamic sparse attention mechanism where the sparsity pattern is determined by the input content** — using hashing, clustering, or learned routing to identify which key-value pairs are most relevant to each query, attending only to those. **Key Approaches** - **Reformer (LSH)**: Locality-Sensitive Hashing groups similar queries and keys into the same bucket. - **Routing Transformer**: Learned routing assigns tokens to clusters, attention within clusters only. - **Clustered Attention**: K-means clustering of queries/keys, attention within clusters. - **Top-$k$ Selection**: Compute approximate attention scores, attend only to top-$k$ keys. **Why It Matters** - **Adaptive**: The sparsity pattern adapts to the input — important dependencies are never missed. - **Better Than Fixed**: Can capture irregular, content-dependent long-range dependencies that fixed patterns miss. - **Challenge**: The routing/hashing overhead must be small enough to justify the attention savings. **Content-Based Sparse Attention** is **attention that finds its own shortcuts** — dynamically discovering which tokens matter most to each query.