feature pyramid from vit
**Feature Pyramid extraction from Vision Transformers** addresses the **fundamental architectural mismatch between the single-scale, columnar output of a standard ViT (which maintains constant spatial resolution throughout all layers) and the multi-scale Feature Pyramid Network (FPN) required by all high-performance object detection and instance segmentation frameworks such as RetinaNet, Faster R-CNN, and Mask R-CNN.**
**The Multi-Scale Requirement**
- **The Detection Pipeline**: Modern object detectors require a hierarchical pyramid of feature maps at multiple spatial resolutions — typically $1/4$, $1/8$, $1/16$, and $1/32$ of the original image resolution. Small objects are detected on high-resolution feature maps, while large objects are detected on coarse, semantically rich feature maps.
- **The CNN Natural Pyramid**: Hierarchical CNNs (ResNet, EfficientNet) naturally produce this pyramid. Each successive stage halves the spatial resolution while doubling the channel depth, creating the exact graduated hierarchy that FPN expects.
- **The ViT Problem**: A standard Vision Transformer (ViT-B/16) splits the image into $16 imes 16$ patches, producing a single sequence of tokens all at $1/16$ resolution. There is no $1/4$, $1/8$, or $1/32$ stage. The output is a flat, single-scale representation completely incompatible with the pyramid paradigm.
**The Three Extraction Strategies**
1. **Simple Feature Map (Naive)**: Reshape the ViT output tokens back into a 2D spatial grid at $1/16$ resolution and use it as a single-scale input. This completely ignores multi-scale requirements and severely degrades small object detection.
2. **Hierarchical ViTs (Swin Transformer)**: Purpose-built architectures like Swin Transformer redesign the ViT to naturally produce a pyramid. Swin uses Patch Merging layers that progressively halve the spatial resolution between stages, automatically generating the $1/4$, $1/8$, $1/16$, and $1/32$ feature maps that FPN demands.
3. **ViTDet (Artificial Pyramid Reconstruction)**: For plain, columnar ViTs (ViT-B, ViT-L, ViT-H) that inherently produce only a single-scale output, ViTDet applies a Simple Feature Pyramid (SFP). The single $1/16$ feature map is processed through parallel branches: transposed convolutions (deconvolutions) upsample it to create the $1/4$ and $1/8$ scales, while max-pooling downsamples it to create the $1/32$ scale. This artificially reconstructs the full pyramid from a flat representation.
**Feature Pyramid from ViT** is **retrofitting a skyscraper with fire escapes** — surgically reconstructing the multi-scale hierarchical structure that object detectors demand from an architecture that was originally designed to see the world at only a single, fixed resolution.