Vision State Space Models (VSSM) are the sequence modeling successors that treat images as flattened sequences and apply linear state-space recurrences to achieve global receptive fields with linear time — by combining state-space layers (such as S4) with convolutional input/output projections, VSSMs process very long vision sequences without the quadratic bottleneck of attention.

What Is a Vision State Space Model?

• Definition: An architecture that views each image as a 1D token stream and feeds it through state-space layers that update an internal hidden state using linear recurrences, followed by output projections that reshape the state into patches.
• Key Feature 1: SSM layers maintain global context through linear time updates, so they do not require sparse or windowed attention.
• Key Feature 2: Input/output convolutions map between 2D patches and the 1D sequence expected by the SSM layer.
• Key Feature 3: Parameterized kernels (e.g., hi-parameterization or power series) control the memory of the recurrence.
• Key Feature 4: VSSMs often surround the state-space block with residual connections and normalization to match transformer-style training.

Why VSSM Matters

• Linear Complexity: Compute grows linearly with sequence length, enabling video or gigapixel images to be processed affordably.
• Global Context: The recurrence inherently mixes all tokens, so even long-range dependencies are captured without explicit attention patterns.
• Robustness: Deterministic recurrences can be more stable than stochastic attention, especially on streaming inputs.
• Hardware Friendliness: State-space layers use matrix-vector products similar to convolutions, making them easy to optimize on chips.
• Complementary: VSSMs can replace only the attention blocks in a hybrid transformer, keeping other components unchanged.

State Space Choices

S4 (Structured State Space):

• Uses parameterized kernels derived from HiPPO matrices for long memory.
• Offers exponential decay that matches both short and long contexts.

Liquid S4:

• Adds gating mechanisms to mix multiple SSMs.
• Improves expressivity with minimal compute overhead.

Kernelized Recurrences:

• Use learned kernels that define the impulse response rather than fixed matrices.
• Provide fine control over temporal decay.

How It Works / Technical Details

Step 1: Flatten the image into a 1D sequence of patch embeddings, feed it through a convolutional projection to match the SSM input dimension, and pass it through the state-space recurrence which updates a hidden state per step.

Step 2: Project the resulting sequence back to tokens, add residual connections, and reshape into spatial patches for downstream layers or heads.

Comparison / Alternatives

Tools & Platforms

• StateSpaceModels repo: Contains S4 and Liquid S4 implementations for vision tasks.
• FlashAttention: Can fuse state-space recurrences with minimal overhead during inference.
• Hugging Face: Some models include state-space encoders as alternatives to attention.
• Profilers: Monitor token throughput to confirm linear scaling gains.

Vision SSMs are the recurrence-based alternative to attention that keeps the entire token stream within reach while staying linear in length — they bring the robustness of signal processing to modern vision architectures.