strided attention

**Strided Attention** is a **sparse attention pattern where each token attends to every $s$-th token in the sequence** — creating a dilated attention pattern that efficiently captures long-range dependencies without computing full $O(N^2)$ attention. **How Does Strided Attention Work?** - **Pattern**: Token $i$ attends to tokens ${i - s, i - 2s, ...}$ (every $s$-th previous token). - **Stride $s$**: Typically $s = sqrt{N}$ so each token attends to $sqrt{N}$ positions. - **Combined**: Often paired with local attention — local captures nearby context, strided captures distant context. - **Paper**: Child et al. (2019, Sparse Transformer). **Why It Matters** - **Long-Range**: Captures dependencies across the full sequence length with only $O(sqrt{N})$ attention per token. - **Complementary**: Combined with local attention, provides both fine-grained local and coarse global context. - **Image Generation**: Originally designed for autoregressive image generation (attending to spatially distant pixels). **Strided Attention** is **dilated convolution for attention** — skipping tokens at regular intervals to efficiently reach across the entire sequence.