gMLP (Gated MLP) is an MLP-based architecture that introduces a gating mechanism to the spatial mixing operation, using a Spatial Gating Unit (SGU) that modulates token interactions through element-wise multiplication of a gated branch with a linearly mixed branch. gMLP achieves competitive performance with Transformers on both NLP and vision tasks by combining the simplicity of MLPs with the expressiveness of multiplicative gating.

Why gMLP Matters in AI/ML: gMLP demonstrated that multiplicative gating can compensate for the lack of attention in MLP-based architectures, closing the gap with Transformers even on tasks previously thought to require attention, such as BERT-level masked language modeling.

• Spatial Gating Unit (SGU) — The SGU splits the hidden representation into two halves: one half is linearly projected across spatial positions (W·Z + b, where W mixes tokens) and the result is element-wise multiplied with the other half; this gating enables input-dependent spatial mixing despite using fixed linear weights • Input-dependent mixing — Unlike MLP-Mixer (purely linear, data-independent spatial mixing) and FNet (fixed FFT), gMLP's multiplicative gate makes the effective spatial mixing data-dependent: the gate values depend on the current input, creating a form of soft, content-based routing • Architecture simplicity — Each gMLP block consists of: (1) LayerNorm, (2) channel expansion MLP (project up), (3) SGU (spatial gating), (4) channel projection MLP (project down), (5) residual connection; no attention, no explicit position encoding • NLP competitiveness — On BERT benchmarks, gMLP matches BERT performance when scaled to similar model sizes, demonstrating that attention is not strictly necessary for strong natural language understanding when replaced with gated spatial mixing • Vision performance — On ImageNet, gMLP matches DeiT (data-efficient ViT) at comparable model sizes and FLOPs, establishing that gated MLPs are a viable alternative to vision transformers for image classification

gMLP bridges the gap between pure MLP architectures and attention-based Transformers through its Spatial Gating Unit, which introduces data-dependent token mixing via multiplicative gating, demonstrating that this simple mechanism is sufficient to match Transformer performance on both vision and language tasks without any attention computation.