All-MLP architectures are the vision model families that replace attention and most convolution blocks with dense token and channel mixing layers - proving that strong image understanding can come from simple matrix multiplication pipelines when patch embeddings, normalization, and training recipes are designed correctly.

What Are All-MLP Architectures?

• Definition: Neural networks that process patch embeddings using multilayer perceptrons for both spatial interaction and feature transformation.
• Core Idea: Separate spatial mixing from channel mixing, then alternate those operations in residual blocks.
• Input Format: Images are split into fixed patches, projected to embeddings, and treated as a token grid.
• Key Property: No explicit self-attention is required to model long range dependencies.

Why All-MLP Architectures Matter

• Simplicity: Fewer primitive operations make implementation and optimization easier.
• Hardware Fit: Dense matmul kernels run efficiently on GPUs, TPUs, and accelerators.
• Scalability: Performance improves with stronger data augmentation and larger pretraining sets.
• Ablation Clarity: Researchers can isolate which gains come from architecture versus training recipe.
• Design Space: They provide a clean baseline for comparing attention, convolution, and hybrid models.

Common Design Patterns

Token Mixer Blocks:

• MLP over the token dimension to exchange spatial information.
• Residual path preserves stable gradients in deep stacks.

Channel Mixer Blocks:

• MLP over channel dimension to learn feature interactions.
• Usually paired with GELU and dropout.

Normalization Strategy:

• LayerNorm, RMSNorm, or affine-only normalization depending on stability targets.

How It Works

Step 1: Patchify image and project each patch to an embedding, then apply token-mixing MLP to propagate spatial context.

Step 2: Apply channel-mixing MLP, residual addition, and normalization repeatedly until a classifier head maps pooled features to labels.

Tools & Platforms

• timm: Includes Mixer, ResMLP, gMLP, and MetaFormer style baselines.
• PyTorch Lightning: Useful for structured ablation of mixers and normalization choices.
• ONNX Runtime: Efficiently deploys matmul heavy models on edge and cloud targets.

All-MLP architectures are a strong proof that good patch pipelines and training discipline can rival more complex vision blocks - they give teams a clean and fast baseline before adding attention or convolution complexity.