fixed attention patterns

**Fixed Attention Patterns** are **predetermined, static sparsity patterns for self-attention** — where the set of positions each token can attend to is defined before training and does not depend on the input content, enabling efficient implementation. **Types of Fixed Patterns** - **Block Diagonal**: Divide sequence into blocks. Each token attends only within its block. - **Dilated/Strided**: Regular stride patterns across the sequence. - **Axial**: Attend along one dimension at a time (for 2D data). - **Global Tokens**: Designate a few tokens as "global" that attend to and are attended by all tokens. - **Combination**: Longformer/BigBird combine local windows + global tokens + random connections. **Why It Matters** - **Predictable**: Fixed patterns enable highly optimized CUDA kernels and hardware-aware implementations. - **Proven**: Longformer and BigBird demonstrate that fixed patterns can match full attention on long document tasks. - **Scalable**: Complexity is $O(N)$ for most fixed patterns (linear in sequence length). **Fixed Attention Patterns** are **the predetermined wiring diagrams for attention** — trading flexibility for efficiency with hand-designed connectivity structures.