crossvit
**CrossViT** is the **dual-branch transformer that processes fine- and coarse-grained patch streams simultaneously and lets them exchange context via cross-attention** — one branch sees small patches for texture while the other sees larger patches for layout, and bi-directional attention ensures both scales collaborate before classification.
**What Is CrossViT?**
- **Definition**: A vision transformer architecture with two parallel encoders: a tiny-patch branch (e.g., 8×8) and a large-patch branch (e.g., 16×16), each with its own attention layers.
- **Key Feature 1**: Cross-attention modules allow the branches to query each other, blending high-resolution cues with low-resolution context.
- **Key Feature 2**: Branch outputs are merged through concatenation or addition before the classifier, preserving multi-scale richness.
- **Key Feature 3**: Each branch can have different depths and channel widths to maintain computational balance.
- **Key Feature 4**: Relative positional biases align tokens across scales.
**Why CrossViT Matters**
- **Scale Robustness**: Small patches catch fine texture while large patches capture object-level structure, helping classification and detection alike.
- **Efficient Fusion**: Rather than building a massive single branch, the model processes two smaller streams in parallel.
- **Transfer Flexibility**: Branch-specific heads allow fine-tuning one branch for a new task while keeping the other frozen.
- **Interpretability**: Attention maps reveal whether decisions rely on detail or layout, aiding visualization.
- **Plugin Friendly**: CrossViT modules can be inserted into existing ViT backbones to add multi-scale reasoning.
**Branch Configurations**
**Balance Strategy**:
- Keep total FLOPs constant by adjusting depth and width per branch.
- Assign more layers to the small-patch branch for detail representation.
**Cross-Attn Frequency**:
- Insert cross-attention every few layers to share information at key intervals.
- Skip early cross-attention to let each branch extract its own features first.
**Hierarchical Merge**:
- Combine branch tokens progressively before final classification to create a fused representation.
**How It Works / Technical Details**
**Step 1**: Each branch computes standard multi-head attention within its patch scale, producing encoded tokens of matching spatial sizes.
**Step 2**: Cross-attention modules treat one branch as queries and the other as keys/values and vice versa, enabling mutual conditioning. The fused tokens then proceed through feed-forward layers and eventual concatenation.
**Comparison / Alternatives**
| Aspect | CrossViT | Pyramid ViT | Single-scale ViT |
|--------|----------|-------------|------------------|
| Scales | Dual fixed | Multi-stage | Single |
| Fusion | Cross-attention | Concatenation/FPN | None |
| Parameter Count | Moderate | Higher | Lowest |
| Applications | Fine+coarse tasks | Detection, segmentation | Classification |
**Tools & Platforms**
- **Hugging Face Transformers**: Contains CrossVitModel and CrossVitForImageClassification.
- **timm**: Implements cross attention layers that can plug into standard ViTs.
- **MMDetection**: Allows CrossViT backbones for detection by exposing feature maps at both scales.
- **Visualization suites**: Tools like Captum reveal cross-attention weights between scales.
CrossViT is **the elegant multi-resolution duet that lets detail and layout sing together without forcing a single branch to be both wide and deep** — it mixes fine texture with anchoring context for resilient visual recognition.