Feature Pyramid extraction from Vision Transformers addresses the fundamental architectural mismatch between the single-scale, columnar output of a standard ViT (which maintains constant spatial resolution throughout all layers) and the multi-scale Feature Pyramid Network (FPN) required by all high-performance object detection and instance segmentation frameworks such as RetinaNet, Faster R-CNN, and Mask R-CNN.

The Multi-Scale Requirement

• The Detection Pipeline: Modern object detectors require a hierarchical pyramid of feature maps at multiple spatial resolutions — typically $1/4$, $1/8$, $1/16$, and $1/32$ of the original image resolution. Small objects are detected on high-resolution feature maps, while large objects are detected on coarse, semantically rich feature maps.
• The CNN Natural Pyramid: Hierarchical CNNs (ResNet, EfficientNet) naturally produce this pyramid. Each successive stage halves the spatial resolution while doubling the channel depth, creating the exact graduated hierarchy that FPN expects.
• The ViT Problem: A standard Vision Transformer (ViT-B/16) splits the image into $16 imes 16$ patches, producing a single sequence of tokens all at $1/16$ resolution. There is no $1/4$, $1/8$, or $1/32$ stage. The output is a flat, single-scale representation completely incompatible with the pyramid paradigm.

The Three Extraction Strategies

1. Simple Feature Map (Naive): Reshape the ViT output tokens back into a 2D spatial grid at $1/16$ resolution and use it as a single-scale input. This completely ignores multi-scale requirements and severely degrades small object detection.

2. Hierarchical ViTs (Swin Transformer): Purpose-built architectures like Swin Transformer redesign the ViT to naturally produce a pyramid. Swin uses Patch Merging layers that progressively halve the spatial resolution between stages, automatically generating the $1/4$, $1/8$, $1/16$, and $1/32$ feature maps that FPN demands.

3. ViTDet (Artificial Pyramid Reconstruction): For plain, columnar ViTs (ViT-B, ViT-L, ViT-H) that inherently produce only a single-scale output, ViTDet applies a Simple Feature Pyramid (SFP). The single $1/16$ feature map is processed through parallel branches: transposed convolutions (deconvolutions) upsample it to create the $1/4$ and $1/8$ scales, while max-pooling downsamples it to create the $1/32$ scale. This artificially reconstructs the full pyramid from a flat representation.

Feature Pyramid from ViT is retrofitting a skyscraper with fire escapes — surgically reconstructing the multi-scale hierarchical structure that object detectors demand from an architecture that was originally designed to see the world at only a single, fixed resolution.