split attention

**Split Attention** is the **attention mechanism used in ResNeSt (Split-Attention Networks)** — which splits feature map channels into cardinal groups and further into radix splits, applying attention across splits within each group to dynamically weight different representations. **How Does Split Attention Work?** - **Cardinal Groups**: Like ResNeXt, divide channels into $K$ cardinal groups. - **Radix Splits**: Within each group, further split into $R$ radix branches (different kernel sizes or transformations). - **Attention**: Apply softmax attention across the $R$ radix splits within each group based on global channel statistics. - **Paper**: Zhang et al. (2020, ResNeSt). **Why It Matters** - **Dynamic Selection**: Adaptively weights different feature representations within each group. - **ResNeSt**: ResNeSt-50 significantly outperforms ResNet-50 and matches ResNet-152 accuracy. - **Downstream**: Strong backbone for detection and segmentation when combined with Feature Pyramid Networks. **Split Attention** is **attention within groups** — dynamically selecting the best representation from multiple radix splits within each cardinal group.