multi-scale vit
**MViT (Multi-Scale Vision Transformer)** is the **pyramidal transformer architecture that progressively reduces spatial resolution while increasing channel depth so the network captures both local details and global context without massive FLOPs** — each stage pools tokens, doubles channels, and applies attention, mimicking how CNN backbones shrink height and width while keeping semantic richness.
**What Is MViT?**
- **Definition**: A multi-stage transformer that alternates between reduction blocks (pooling or strided attention) and transformer blocks, forming a feature pyramid similar to ResNet.
- **Key Feature 1**: Early stages preserve high spatial resolution for fine-grained details by using small strides.
- **Key Feature 2**: Later stages pool aggressively, giving attention blocks a global view with fewer tokens.
- **Key Feature 3**: Channel dimensions expand to compensate for the loss of spatial information, keeping representational capacity consistent.
- **Key Feature 4**: Positional encodings and relative embeddings adjust per stage to reflect changing resolution.
**Why MViT Matters**
- **Multi-Resolution Understanding**: Combines high-resolution texture with low-resolution semantics, crucial for detection and segmentation.
- **Efficient Computation**: Each stage reduces the token count, so later layers cost far less despite being deeper.
- **Compatibility with FPN**: Its pyramidal outputs plug directly into necks like PANet or BiFPN for downstream tasks.
- **Robust to Scale Variations**: Processing the same scene at multiple scales helps the model handle objects of diverse sizes.
- **Transfer Learning Friendly**: Resembles CNN stage structure, so pretrained weights from dense networks can inspire initialization.
**Stage Breakdown**
**Stage 1**:
- Operates at input resolution with small patch embeddings (e.g., 4×4) and low channel count.
- Focuses on texture and edge detection.
**Stage 2-3**:
- Use strided attention or pooling to reduce spatial size by roughly half each time while doubling channels.
- Balance cost between localization and context.
**Stage 4**:
- Last stage sees a handful of tokens and captures the global scene layout for classification or detection heads.
**How It Works / Technical Details**
**Step 1**: Each stage applies a token merging or pooling block that reduces height and width while projecting tokens to higher dimension.
**Step 2**: Following the reduction, standard transformer layers with attention and feed-forward networks operate on the smaller token set, and the outputs feed into the next stage.
**Comparison / Alternatives**
| Aspect | MViT | Single-Scale ViT | Swin / Pyramid ViT |
|--------|------|------------------|-------------------|
| Token Count | Decreases per stage | Constant | Decreases via windows |
| Semantic Pyramid | Native | Derived via pooling | Derived via shift/windows |
| FLOPs | Moderate | High (dense) | Moderate |
| Downstream Ready | Yes (FPN) | Needs neck | Yes |
**Tools & Platforms**
- **Hugging Face**: Provides pretrained MViT weights and configs for classification and detection.
- **Detectron2 / MMDetection**: Include MViT backbones for object detection and video understanding.
- **PyTorch Lightning**: Templates for stage-wise transformer training with MViT blocks.
- **Weights & Biases**: Tracks per-stage resolution changes and ensures no stage becomes a bottleneck.
MViT is **the stage-wise transformer design that inherits the best traits of CNN pyramids and ViT expressivity** — it compresses tokens gradually so the network sees local detail and global layout without blowing computation at any single stage.