MLP-Mixer is an architecture for computer vision that replaces both convolutions and self-attention with pure multi-layer perceptrons (MLPs), demonstrating that competitive image classification performance can be achieved using only matrix multiplications and non-linearities applied alternately across spatial locations (token-mixing) and feature channels (channel-mixing). Introduced by Tolstikhin et al. (2021), MLP-Mixer challenged the necessity of both convolutional inductive biases and attention mechanisms.

Why MLP-Mixer Matters in AI/ML: MLP-Mixer demonstrated that neither convolutions nor attention are necessary for strong visual representation learning, suggesting that the key ingredients for modern vision models are sufficient data, scale, and simple token interaction mechanisms.

• Dual MLP structure — Each Mixer layer applies two MLPs sequentially: (1) token-mixing MLP operates across spatial patches (transposed input: features × patches → MLP → features × patches), mixing information between spatial locations; (2) channel-mixing MLP operates across features independently per patch • Patch embedding — Input images are divided into non-overlapping patches (typically 16×16 or 32×32) and linearly projected to a fixed embedding dimension, identical to the ViT patch embedding; this creates a sequence of N = (H×W)/P² patch tokens • No position encoding — MLP-Mixer's token-mixing MLP implicitly learns position-dependent interactions through its weight matrix (which has shape N×N for N patches), encoding spatial relationships in the learned weights without explicit positional encodings • Fixed spatial resolution — Unlike attention (which adapts to any sequence length), the token-mixing MLP has fixed-size weight matrices tied to the number of patches, meaning MLP-Mixer cannot handle variable-resolution inputs without modification • Data efficiency tradeoff — MLP-Mixer requires large datasets (JFT-300M) to match ViT/CNN performance; on ImageNet-1K alone, it underperforms comparably-sized ViTs and ResNets, suggesting that the lack of inductive biases requires more data to compensate

MLP-Mixer is a paradigm-challenging architecture demonstrating that pure MLPs with alternating spatial and channel mixing achieve competitive vision performance without convolutions or attention, revealing that the essential ingredient for visual representation learning is not architectural inductive bias but rather sufficient data and scale to learn spatial relationships from scratch.