gmlp (gated mlp)
**gMLP (Gated MLP)** is an MLP-based architecture that introduces a gating mechanism to the spatial mixing operation, using a Spatial Gating Unit (SGU) that modulates token interactions through element-wise multiplication of a gated branch with a linearly mixed branch. gMLP achieves competitive performance with Transformers on both NLP and vision tasks by combining the simplicity of MLPs with the expressiveness of multiplicative gating.
**Why gMLP Matters in AI/ML:**
gMLP demonstrated that **multiplicative gating can compensate for the lack of attention** in MLP-based architectures, closing the gap with Transformers even on tasks previously thought to require attention, such as BERT-level masked language modeling.
• **Spatial Gating Unit (SGU)** — The SGU splits the hidden representation into two halves: one half is linearly projected across spatial positions (W·Z + b, where W mixes tokens) and the result is element-wise multiplied with the other half; this gating enables input-dependent spatial mixing despite using fixed linear weights
• **Input-dependent mixing** — Unlike MLP-Mixer (purely linear, data-independent spatial mixing) and FNet (fixed FFT), gMLP's multiplicative gate makes the effective spatial mixing data-dependent: the gate values depend on the current input, creating a form of soft, content-based routing
• **Architecture simplicity** — Each gMLP block consists of: (1) LayerNorm, (2) channel expansion MLP (project up), (3) SGU (spatial gating), (4) channel projection MLP (project down), (5) residual connection; no attention, no explicit position encoding
• **NLP competitiveness** — On BERT benchmarks, gMLP matches BERT performance when scaled to similar model sizes, demonstrating that attention is not strictly necessary for strong natural language understanding when replaced with gated spatial mixing
• **Vision performance** — On ImageNet, gMLP matches DeiT (data-efficient ViT) at comparable model sizes and FLOPs, establishing that gated MLPs are a viable alternative to vision transformers for image classification
| Property | gMLP | MLP-Mixer | Transformer |
|----------|------|-----------|-------------|
| Spatial Mixing | Gated linear | Linear MLP | Self-attention |
| Data Dependence | Partial (via gating) | None | Full |
| NLP Performance | ≈ BERT | Not competitive | Baseline |
| Vision Performance | ≈ DeiT | Below ViT | Baseline |
| Parameters | Similar | Similar | Similar |
| Complexity | O(N·d²) | O(N·d²) | O(N²·d) |
**gMLP bridges the gap between pure MLP architectures and attention-based Transformers through its Spatial Gating Unit, which introduces data-dependent token mixing via multiplicative gating, demonstrating that this simple mechanism is sufficient to match Transformer performance on both vision and language tasks without any attention computation.**