vision mamba

**Vision Mamba (Vim)** is a **revolutionary computer vision backbone architecture that completely replaces the computationally expensive Quadratic Self-Attention mechanism of the Vision Transformer (ViT) with the brutally efficient Selective State Space Model (SSM) from the Mamba language model — achieving competitive or superior image classification accuracy while scaling linearly with image resolution instead of quadratically.** **The Quadratic Attention Bottleneck** - **The ViT Problem**: A standard Vision Transformer splits an image into $N$ patches and computes pairwise Self-Attention across all $N$ tokens. The computational cost scales as $O(N^2)$. For a $224 imes 224$ image with $16 imes 16$ patches, $N = 196$ and the cost is manageable. For a high-resolution $1024 imes 1024$ medical scan, $N = 4096$ and the attention matrix explodes to $16.7$ million entries per head, consuming catastrophic GPU memory. - **The Goal**: Achieve the global receptive field of a Transformer (unlike the strictly local receptive field of CNNs) while maintaining linear $O(N)$ computational complexity. **The State Space Model Backbone** - **Patch Tokenization**: Identical to ViT, the input image is split into non-overlapping $16 imes 16$ patches and linearly projected into a sequence of embedding vectors. - **Bidirectional Scanning**: A 1D sequence model like Mamba naturally reads tokens left-to-right. Images, however, are inherently 2D. Vision Mamba solves this by processing the patch sequence in multiple scan orders — forward, backward, and potentially cross-scan (row-major and column-major). This bidirectional processing ensures that every patch can aggregate spatial context from all four cardinal directions, reconstructing 2D spatial awareness from a 1D sequential model. - **Selective State Space Layers**: Instead of computing a massive $N imes N$ attention matrix, each Mamba layer maintains a compact, continuously evolving hidden state vector. Each incoming patch token selectively updates this compressed state, dynamically choosing what information to remember and what to forget based on the content of the current patch. **The Performance Profile** Vision Mamba demonstrates comparable accuracy to DeiT (Data-efficient Image Transformers) on ImageNet classification while consuming significantly less GPU memory and achieving faster inference throughput on high-resolution inputs. The linear scaling makes it particularly attractive for dense prediction tasks (semantic segmentation, object detection) on large images where ViT's quadratic cost becomes prohibitive. **Vision Mamba** is **linear-complexity global vision** — granting an image recognition model the power to see across the entire photograph simultaneously without paying the catastrophic quadratic tax that cripples standard Vision Transformers at high resolution.