CrossViT is a dual-branch vision Transformer that processes image patches at two different scales (small patches for fine-grained detail, large patches for global context) and fuses information between branches through cross-attention using the CLS tokens as bridges. This multi-scale design enables the model to capture both local details and global structure simultaneously while maintaining computational efficiency through the compact cross-attention mechanism.

Why CrossViT Matters in AI/ML: CrossViT introduced the dual-branch multi-scale paradigm for vision Transformers, demonstrating that processing patches at multiple resolutions with cross-scale information fusion outperforms single-scale processing, inspiring subsequent multi-scale vision architectures.

• Dual-branch architecture — Two ViT branches process the same image at different patch sizes: a "large" branch with large patches (e.g., 16×16, fewer tokens) for global context and a "small" branch with small patches (e.g., 12×12 or 8×8, more tokens) for local detail • CLS token cross-attention — Information exchange between branches occurs through the CLS tokens: each branch's CLS token cross-attends to the other branch's patch tokens, aggregating complementary scale information that is then broadcast back to its own branch • Efficient cross-scale fusion — Instead of full cross-attention between all tokens of both branches (which would be expensive), using only the CLS token as an information bottleneck makes the cross-attention cost negligible: O(N_small + N_large) rather than O(N_small × N_large) • Multi-scale feature extraction — The small-patch branch captures fine textures and edges at high spatial resolution while the large-patch branch captures global shapes and semantic structures, and the CLS cross-attention ensures both representations benefit from the other's perspective • Asymmetric branch design — The branches can have different depths, widths, and number of heads, with the large-patch branch typically being wider/deeper (faster per token) and the small-patch branch being narrower/shallower (more tokens to process)

CrossViT pioneered the dual-branch multi-scale approach to vision Transformers, demonstrating that processing images at two patch resolutions with efficient CLS-token cross-attention fusion outperforms single-scale ViTs by leveraging complementary fine-grained and coarse-grained visual representations, inspiring the broader multi-scale vision Transformer paradigm.