sliding window attention

**Sliding Window and Local Sparse Attention** are **attention patterns restricting each token to attend only to nearby context within fixed window size — reducing attention complexity from quadratic O(n²) to linear O(n·w) enabling efficient processing of very long documents (100K+ tokens) on single GPUs**. **Sliding Window Attention Mechanism:** - **Window Definition**: each token at position i attends only to tokens in [i-w, i+w] range where w is window size (512-2048 typical) - **Attention Matrix Structure**: creating banded diagonal matrix instead of full matrix — only w×n non-zero entries instead of n² entries - **Computational Complexity**: reducing FLOPS from O(n²·d) to O(n·w·d) and memory from O(n²) to O(n·w) — linear in sequence length - **Implementation**: using efficient kernels (NVIDIA FlashAttention) with row-wise masking — only 2-3x slower than single-head attention despite sparsity - **Receptive Field**: w=512 provides receptive field enabling local reasoning within paragraph or sentence scope **Local Attention Patterns:** - **Fixed Window**: uniform window size across all positions — simplest, best for causal (left-to-right only) or bidirectional attention - **Dilated Window**: attending to every k-th token in extended range (e.g., positions [i-2w, i, step=k]) — captures longer range dependencies - **Strided Attention**: combining fine-grained local (w=128) with coarse-grained remote (stride=4, attending to every 4th token) — 2-level hierarchy - **Centered Window**: attending to neighbors symmetrically around position i — useful for document encoding (BERT-style) where future context available **Longformer Architecture:** - **Hybrid Approach**: combining local windowed attention with task-specific global attention tokens — key tokens (CLS, document summary markers) attend globally - **Configuration**: local window size w=512, 4 attention heads use global attention on special tokens — remaining 8 heads use sliding window - **Complexity**: O(n·w) local + O(n·g) global where g is number of global tokens (g<