convolution-free vision models
**Convolution-Free Vision Models** are the **architectures that rely solely on attention, MLPs, or state-space recurrences without traditional convolutional kernels, proving that transformers and MLP mixers can still capture image structure** — these models often include positional encodings, gating, or token mixing layers to replace the inductive bias provided by convolutions.
**What Are Convolution-Free Vision Models?**
- **Definition**: Networks that avoid convolution kernels altogether, instead using attention, MLP mixing, or recurrent mechanisms to aggregate spatial information.
- **Key Feature 1**: Positional encodings or learned tokens supply spatial context otherwise embedded in convolutional shifts.
- **Key Feature 2**: Token mixers like MLP-Mixer or gMLP use dense layers to mix patch representations.
- **Key Feature 3**: Many still incorporate gating or token shuffling to mimic local connectivity.
- **Key Feature 4**: Some hybridize with lightweight convolutions only in the embedding layer for initial patch projection.
**Why They Matter**
- **Research Value**: Demonstrate that the convolutional inductive bias is not strictly necessary for strong visual representation learning.
- **Simplified Architecture**: Reduces dependency on optimized convolution kernels, which can be beneficial for certain hardware platforms.
- **Transferability**: Their general mixing layers often transfer well to modalities beyond vision.
- **Flexibility**: Easily combine with other modalities (text, audio) thanks to the absence of domain-specific convolution rules.
- **Innovation**: Inspires new building blocks such as token mixers, structured MLPs, and implicit position modeling.
**Model Families**
**ViT / Transformer**:
- Pure attention with patch embeddings and learnable class tokens.
- Relies on positional embeddings to encode spatial structure.
**MLP Mixers / gMLP**:
- Use alternating token-mixing and channel-mixing MLPs.
- Introduce gating (e.g., spatial gating units) to direct flows.
**State-Space Models**:
- Flatten patches into sequences and apply linear recurrences (VSSM, RetNet, RWKV).
- Provide long-range modeling without convolution.
**How It Works / Technical Details**
**Step 1**: Convert the image into patch embeddings via a linear projection; optionally add sinusoidal or learned positional embeddings.
**Step 2**: Run the chosen mix/attention blocks (transformer layers, MLP mixers, state-space recurrences) across the sequence, optionally interleaving gating or normalization layers to preserve stability.
**Comparison / Alternatives**
| Aspect | Convolution-Free | ConvNet | Hybrid (Conv + Attn) |
|--------|------------------|---------|----------------------|
| Inductive Bias | None (learned) | Strong (local) | Moderate
| Modality Flexibility | High | Medium | Medium
| Hardware | Matmul-heavy | Convolution-friendly | Mixed
| Research Impact | High (agnostic) | Classic | Transitional
**Tools & Platforms**
- **timm**: Houses ViT, MLP-Mixer, gMLP, and similar convolution-free implementations.
- **Hugging Face**: Hosts pre-trained convolution-free backbones for classification and vision-language tasks.
- **TVM / Triton**: Optimize matmul-heavy pipelines that replace convolution.
- **Visualization**: Plot attention or mixing weights to ensure spatial coherence is still captured.
Convolution-free vision models are **the experimental proof that pure mixing and attention can rival convolutional hierarchies** — they push the boundaries of what purely learned inductive biases can achieve without manual kernel design.