coat (co-scale conv-attentional image transformers)
**CoAT (Co-Scale Conv-Attentional Image Transformers)** is a hierarchical vision Transformer that introduces co-scale attention—a mechanism for exchanging information between feature representations at different spatial scales through cross-attention—combined with convolutional relative position encoding within each scale. CoAT processes images at multiple resolutions simultaneously and fuses multi-scale information through learned cross-scale attention, enabling rich representations that capture both fine details and global context.
**Why CoAT Matters in AI/ML:**
CoAT addresses the **multi-scale information flow problem** in hierarchical vision Transformers, enabling explicit cross-scale feature interaction that strengthens both fine-grained and coarse-grained representations beyond what independent per-scale processing or simple feature pyramids achieve.
• **Co-scale attention mechanism** — Feature maps at different scales exchange information through cross-attention: high-resolution features query low-resolution features (obtaining global context) and low-resolution features query high-resolution features (obtaining fine details), creating bidirectional multi-scale interaction
• **Factorized attention** — CoAT factorizes attention into serial and parallel components: serial blocks process each scale independently with self-attention; parallel blocks compute cross-attention between scales, enabling efficient multi-scale processing
• **Convolutional relative position encoding** — Position information is encoded through depth-wise convolutions applied to the value projections, providing translation-equivariant, content-independent positional signals without explicit position embeddings
• **Multi-scale feature fusion** — Unlike Swin/PVT (which produce multi-scale features but process each scale independently), CoAT actively fuses information across scales during processing, producing more coherent multi-scale representations
• **Dense prediction strength** — The explicit cross-scale attention makes CoAT particularly strong for detection and segmentation tasks where relating fine-grained details to global scene context is critical
| Component | CoAT | Swin | PVT | CrossViT |
|-----------|------|------|-----|----------|
| Multi-Scale | Cross-scale attention | Independent scales | Independent scales | Dual-branch cross-attn |
| Scale Interaction | Bidirectional cross-attn | Shifted windows | None (per-stage) | Cross-attention tokens |
| Position Encoding | Conv relative | Relative bias | Learned absolute/conv | Learned absolute |
| Hierarchy | 4 stages | 4 stages | 4 stages | 2 branches |
| Cross-Scale Flow | Explicit, bidirectional | None (sequential) | None (sequential) | Limited (CLS token) |
**CoAT advances hierarchical vision Transformers by introducing explicit bidirectional cross-scale attention that enables rich multi-scale feature interaction during processing—not just at the output—ensuring that representations at every scale benefit from both fine-grained detail and global context, producing superior features for dense prediction tasks.**