Convolution-Free Vision Models are the architectures that rely solely on attention, MLPs, or state-space recurrences without traditional convolutional kernels, proving that transformers and MLP mixers can still capture image structure — these models often include positional encodings, gating, or token mixing layers to replace the inductive bias provided by convolutions.

What Are Convolution-Free Vision Models?

• Definition: Networks that avoid convolution kernels altogether, instead using attention, MLP mixing, or recurrent mechanisms to aggregate spatial information.
• Key Feature 1: Positional encodings or learned tokens supply spatial context otherwise embedded in convolutional shifts.
• Key Feature 2: Token mixers like MLP-Mixer or gMLP use dense layers to mix patch representations.
• Key Feature 3: Many still incorporate gating or token shuffling to mimic local connectivity.
• Key Feature 4: Some hybridize with lightweight convolutions only in the embedding layer for initial patch projection.

Why They Matter

• Research Value: Demonstrate that the convolutional inductive bias is not strictly necessary for strong visual representation learning.
• Simplified Architecture: Reduces dependency on optimized convolution kernels, which can be beneficial for certain hardware platforms.
• Transferability: Their general mixing layers often transfer well to modalities beyond vision.
• Flexibility: Easily combine with other modalities (text, audio) thanks to the absence of domain-specific convolution rules.
• Innovation: Inspires new building blocks such as token mixers, structured MLPs, and implicit position modeling.

Model Families

ViT / Transformer:

• Pure attention with patch embeddings and learnable class tokens.
• Relies on positional embeddings to encode spatial structure.

MLP Mixers / gMLP:

• Use alternating token-mixing and channel-mixing MLPs.
• Introduce gating (e.g., spatial gating units) to direct flows.

State-Space Models:

• Flatten patches into sequences and apply linear recurrences (VSSM, RetNet, RWKV).
• Provide long-range modeling without convolution.

How It Works / Technical Details

Step 1: Convert the image into patch embeddings via a linear projection; optionally add sinusoidal or learned positional embeddings.

Step 2: Run the chosen mix/attention blocks (transformer layers, MLP mixers, state-space recurrences) across the sequence, optionally interleaving gating or normalization layers to preserve stability.

Comparison / Alternatives

Tools & Platforms

• timm: Houses ViT, MLP-Mixer, gMLP, and similar convolution-free implementations.
• Hugging Face: Hosts pre-trained convolution-free backbones for classification and vision-language tasks.
• TVM / Triton: Optimize matmul-heavy pipelines that replace convolution.
• Visualization: Plot attention or mixing weights to ensure spatial coherence is still captured.

Convolution-free vision models are the experimental proof that pure mixing and attention can rival convolutional hierarchies — they push the boundaries of what purely learned inductive biases can achieve without manual kernel design.