axial attention

**Axial Attention** is a **factored attention mechanism that decomposes 2D self-attention into two sequential 1D attention operations** — first along the height axis, then along the width axis, reducing complexity from $O(N^2)$ to $O(N sqrt{N})$. **How Does Axial Attention Work?** - **Height Attention**: Each position attends to all positions in its column. - **Width Attention**: Each position then attends to all positions in its row. - **Sequential**: Apply height attention, then width attention (or vice versa). - **Position Encoding**: Relative position embeddings added to queries and keys. - **Paper**: Ho et al. (2019), Wang et al. (2020, Axial-DeepLab). **Why It Matters** - **Scalability**: Enables self-attention on high-resolution images (512×512 and above). - **Segmentation**: Axial-DeepLab achieves strong panoptic segmentation results. - **Image Generation**: Used in efficient attention for image generation models. **Axial Attention** is **2D attention factored into 1D** — decomposing full spatial attention into efficient row-then-column operations.