all-mlp architectures

**All-MLP architectures** are the **vision model families that replace attention and most convolution blocks with dense token and channel mixing layers** - proving that strong image understanding can come from simple matrix multiplication pipelines when patch embeddings, normalization, and training recipes are designed correctly. **What Are All-MLP Architectures?** - **Definition**: Neural networks that process patch embeddings using multilayer perceptrons for both spatial interaction and feature transformation. - **Core Idea**: Separate spatial mixing from channel mixing, then alternate those operations in residual blocks. - **Input Format**: Images are split into fixed patches, projected to embeddings, and treated as a token grid. - **Key Property**: No explicit self-attention is required to model long range dependencies. **Why All-MLP Architectures Matter** - **Simplicity**: Fewer primitive operations make implementation and optimization easier. - **Hardware Fit**: Dense matmul kernels run efficiently on GPUs, TPUs, and accelerators. - **Scalability**: Performance improves with stronger data augmentation and larger pretraining sets. - **Ablation Clarity**: Researchers can isolate which gains come from architecture versus training recipe. - **Design Space**: They provide a clean baseline for comparing attention, convolution, and hybrid models. **Common Design Patterns** **Token Mixer Blocks**: - MLP over the token dimension to exchange spatial information. - Residual path preserves stable gradients in deep stacks. **Channel Mixer Blocks**: - MLP over channel dimension to learn feature interactions. - Usually paired with GELU and dropout. **Normalization Strategy**: - LayerNorm, RMSNorm, or affine-only normalization depending on stability targets. **How It Works** **Step 1**: Patchify image and project each patch to an embedding, then apply token-mixing MLP to propagate spatial context. **Step 2**: Apply channel-mixing MLP, residual addition, and normalization repeatedly until a classifier head maps pooled features to labels. **Tools & Platforms** - **timm**: Includes Mixer, ResMLP, gMLP, and MetaFormer style baselines. - **PyTorch Lightning**: Useful for structured ablation of mixers and normalization choices. - **ONNX Runtime**: Efficiently deploys matmul heavy models on edge and cloud targets. All-MLP architectures are **a strong proof that good patch pipelines and training discipline can rival more complex vision blocks** - they give teams a clean and fast baseline before adding attention or convolution complexity.