attention mechanism hierarchical

**Hierarchical Attention** is an **attention mechanism that operates at multiple levels of granularity** — first computing attention within local groups (words, patches, tokens), then computing attention over group-level representations, creating a multi-scale attention hierarchy. **Common Hierarchical Patterns** - **HAT**: Word-level attention → sentence-level attention → document-level attention. - **Swin Transformer**: Window-level attention → shifted window (inter-window communication). - **HiP Attention**: Hierarchical token pooling with attention at each level. - **Nested Transformers**: Attention within regions, then attention across regions. **Why It Matters** - **Long Sequences**: Handles very long sequences (documents, high-res images) by processing locally first, then globally. - **Efficiency**: $O(N cdot k)$ where $k$ is the local group size, vs. $O(N^2)$ for global attention. - **Multi-Scale**: Naturally captures both fine-grained local patterns and coarse global patterns. **Hierarchical Attention** is **zoom-in-zoom-out attention** — processing information at multiple scales from local details to global summaries.