vision state space models
**Vision State Space Models (VSSM)** are the **sequence modeling successors that treat images as flattened sequences and apply linear state-space recurrences to achieve global receptive fields with linear time** — by combining state-space layers (such as S4) with convolutional input/output projections, VSSMs process very long vision sequences without the quadratic bottleneck of attention.
**What Is a Vision State Space Model?**
- **Definition**: An architecture that views each image as a 1D token stream and feeds it through state-space layers that update an internal hidden state using linear recurrences, followed by output projections that reshape the state into patches.
- **Key Feature 1**: SSM layers maintain global context through linear time updates, so they do not require sparse or windowed attention.
- **Key Feature 2**: Input/output convolutions map between 2D patches and the 1D sequence expected by the SSM layer.
- **Key Feature 3**: Parameterized kernels (e.g., hi-parameterization or power series) control the memory of the recurrence.
- **Key Feature 4**: VSSMs often surround the state-space block with residual connections and normalization to match transformer-style training.
**Why VSSM Matters**
- **Linear Complexity**: Compute grows linearly with sequence length, enabling video or gigapixel images to be processed affordably.
- **Global Context**: The recurrence inherently mixes all tokens, so even long-range dependencies are captured without explicit attention patterns.
- **Robustness**: Deterministic recurrences can be more stable than stochastic attention, especially on streaming inputs.
- **Hardware Friendliness**: State-space layers use matrix-vector products similar to convolutions, making them easy to optimize on chips.
- **Complementary**: VSSMs can replace only the attention blocks in a hybrid transformer, keeping other components unchanged.
**State Space Choices**
**S4 (Structured State Space)**:
- Uses parameterized kernels derived from HiPPO matrices for long memory.
- Offers exponential decay that matches both short and long contexts.
**Liquid S4**:
- Adds gating mechanisms to mix multiple SSMs.
- Improves expressivity with minimal compute overhead.
**Kernelized Recurrences**:
- Use learned kernels that define the impulse response rather than fixed matrices.
- Provide fine control over temporal decay.
**How It Works / Technical Details**
**Step 1**: Flatten the image into a 1D sequence of patch embeddings, feed it through a convolutional projection to match the SSM input dimension, and pass it through the state-space recurrence which updates a hidden state per step.
**Step 2**: Project the resulting sequence back to tokens, add residual connections, and reshape into spatial patches for downstream layers or heads.
**Comparison / Alternatives**
| Aspect | VSSM | Linear Attention | Standard ViT |
|--------|------|------------------|--------------|
| Complexity | O(N) | O(N) | O(N^2)
| Global Context | Yes | Yes | Yes
| Streaming | Excellent | Excellent | Limited
| Implementation | More novel | Medium | Standard
**Tools & Platforms**
- **StateSpaceModels repo**: Contains S4 and Liquid S4 implementations for vision tasks.
- **FlashAttention**: Can fuse state-space recurrences with minimal overhead during inference.
- **Hugging Face**: Some models include state-space encoders as alternatives to attention.
- **Profilers**: Monitor token throughput to confirm linear scaling gains.
Vision SSMs are **the recurrence-based alternative to attention that keeps the entire token stream within reach while staying linear in length** — they bring the robustness of signal processing to modern vision architectures.