deformable attention

**Deformable Attention** is an **attention mechanism that attends to a small set of key sampling points around a reference point** — instead of attending to all spatial positions, reducing the $O(N^2)$ complexity of full attention to $O(N cdot K)$ where $K$ is the number of sampling points. **How Does Deformable Attention Work?** - **Reference Points**: Each query has a reference point (e.g., grid position or predicted object center). - **Sampling Offsets**: $K$ learnable offsets from the reference point (typically $K = 4-8$). - **Attention Weights**: Learned attention weights for each of the $K$ sampling points. - **Multi-Scale**: Can sample from multiple feature map scales simultaneously. - **Paper**: Zhu et al., "Deformable DETR" (2021). **Why It Matters** - **Efficiency**: $O(N cdot K)$ vs. $O(N^2)$ for full attention -> enables high-resolution feature maps. - **DETR Acceleration**: Deformable DETR converges 10× faster than vanilla DETR. - **Detection Standard**: The standard attention mechanism in modern detection transformers. **Deformable Attention** is **attention that samples smartly** — attending to a few learnable positions instead of everything, making transformer detection practical.