crossvit
**CrossViT** is a dual-branch vision Transformer that processes image patches at two different scales (small patches for fine-grained detail, large patches for global context) and fuses information between branches through cross-attention using the CLS tokens as bridges. This multi-scale design enables the model to capture both local details and global structure simultaneously while maintaining computational efficiency through the compact cross-attention mechanism.
**Why CrossViT Matters in AI/ML:**
CrossViT introduced the **dual-branch multi-scale paradigm** for vision Transformers, demonstrating that processing patches at multiple resolutions with cross-scale information fusion outperforms single-scale processing, inspiring subsequent multi-scale vision architectures.
• **Dual-branch architecture** — Two ViT branches process the same image at different patch sizes: a "large" branch with large patches (e.g., 16×16, fewer tokens) for global context and a "small" branch with small patches (e.g., 12×12 or 8×8, more tokens) for local detail
• **CLS token cross-attention** — Information exchange between branches occurs through the CLS tokens: each branch's CLS token cross-attends to the other branch's patch tokens, aggregating complementary scale information that is then broadcast back to its own branch
• **Efficient cross-scale fusion** — Instead of full cross-attention between all tokens of both branches (which would be expensive), using only the CLS token as an information bottleneck makes the cross-attention cost negligible: O(N_small + N_large) rather than O(N_small × N_large)
• **Multi-scale feature extraction** — The small-patch branch captures fine textures and edges at high spatial resolution while the large-patch branch captures global shapes and semantic structures, and the CLS cross-attention ensures both representations benefit from the other's perspective
• **Asymmetric branch design** — The branches can have different depths, widths, and number of heads, with the large-patch branch typically being wider/deeper (faster per token) and the small-patch branch being narrower/shallower (more tokens to process)
| Branch | Patch Size | Tokens (224²) | Detail Level | Role |
|--------|-----------|---------------|-------------|------|
| Large | 16×16 | 196 | Coarse, global | Semantic structure |
| Small | 12×12 | 361 | Fine, local | Texture, edges |
| Cross-Attention | CLS ↔ patches | 1 × (196 or 361) | Inter-scale | Fusion bridge |
| Fused Output | Both CLS tokens | 2 | Combined | Final classification |
**CrossViT pioneered the dual-branch multi-scale approach to vision Transformers, demonstrating that processing images at two patch resolutions with efficient CLS-token cross-attention fusion outperforms single-scale ViTs by leveraging complementary fine-grained and coarse-grained visual representations, inspiring the broader multi-scale vision Transformer paradigm.**