patch merging

**Patch Merging** is a **downsampling operation in Vision Transformers that reduces the number of tokens by merging adjacent patches** — similar to strided convolution in CNNs, creating a hierarchical representation with progressively fewer, richer tokens. **How Does Patch Merging Work?** - **Group**: Take 2×2 groups of adjacent tokens (4 tokens per group). - **Concatenate**: Concatenate their features along the channel dimension ($C → 4C$). - **Project**: Linear projection to reduce channels ($4C → 2C$). - **Result**: Spatial resolution halved (H/2 × W/2), channels doubled ($2C$). - **Used In**: Swin Transformer, Twins, PVT. **Why It Matters** - **Hierarchical ViT**: Enables ViTs to have a multi-scale, pyramid-like structure similar to CNNs. - **Dense Prediction**: The multi-scale feature maps are essential for detection and segmentation. - **Efficiency**: Fewer tokens at later stages -> reduced attention computation. **Patch Merging** is **pooling for Vision Transformers** — creating a multi-resolution feature hierarchy by progressively combining adjacent tokens.