cait

**CaiT (Class-Attention in Image Transformers)** is a **carefully re-engineered Vision Transformer architecture specifically designed to enable extremely deep networks (40+ layers) by surgically separating the feature extraction phase (Self-Attention among image patches) from the classification aggregation phase (Class-Attention between the CLS token and the patch tokens) into two completely distinct, sequential processing stages.** **The Depth Problem in Standard ViTs** - **The CLS Token Interference**: In a standard ViT, the learnable CLS (classification) token is concatenated to the patch token sequence from the very first layer. It participates in every single Self-Attention computation throughout the entire depth of the network. - **The Degradation**: As the network gets deeper (beyond 12-24 layers), the CLS token's constant participation in the patch-level Self-Attention creates a parasitic interference loop. The CLS token simultaneously tries to aggregate a global summary while also influencing the local patch feature representations through its attention weights. This dual role destabilizes training and causes severe performance saturation in very deep ViTs. **The CaiT Two-Stage Architecture** CaiT cleanly resolves this by splitting the network into two distinct phases: 1. **Phase 1 — Self-Attention Layers (SA, Layers 1 to $L_{SA}$)**: Only the image patch tokens participate. The CLS token is completely absent. For 36+ layers, the patches freely refine their local and global feature representations through standard Multi-Head Self-Attention without any interference from a classification-oriented token. 2. **Phase 2 — Class-Attention Layers (CA, Layers $L_{SA}+1$ to $L_{SA}+2$)**: The CLS token is injected for the first time. In these final 2 layers, a modified attention mechanism is applied: the CLS token attends to all patch tokens (reading their refined features), but the patch tokens do not attend to the CLS token and do not attend to each other. The CLS token becomes a pure, focused aggregator. **The LayerScale Innovation** CaiT also introduced LayerScale — multiplying each residual branch output by a learnable, per-channel scalar initialized to a very small value ($10^{-4}$). This prevents the residual connections from dominating the signal in the early training phase and enables stable optimization of networks exceeding 36 layers deep. **CaiT** is **delegated summarization** — refusing to let the executive summary token participate in the chaotic factory-floor feature extraction, instead forcing it to wait silently in the boardroom until all the refined reports arrive for final aggregation.