gmlp for vision

**gMLP** is the **gated MLP architecture that injects spatial interaction through a Spatial Gating Unit while keeping the model attention free** - it multiplies one feature branch by a learned spatial projection of another branch, creating content-aware modulation without softmax attention. **What Is gMLP?** - **Definition**: An MLP based block that splits channels, processes one half through a spatial projection, and gates the other half. - **Spatial Gating Unit**: Central mechanism that enables token level interaction across sequence positions. - **Residual Design**: Standard residual wrappers keep training stable in deeper stacks. - **Flexibility**: Can be used in pure all-MLP backbones or hybridized with convolution and attention blocks. **Why gMLP Matters** - **Content Modulation**: Gating introduces adaptive behavior beyond plain linear token mixing. - **Lower Overhead**: Avoids quadratic attention maps and reduces memory pressure. - **Strong Baseline**: Competitive performance in classification with tuned recipes. - **Hybrid Utility**: Useful as a drop-in block for efficient backbones. - **Research Value**: Helps isolate the benefit of gating versus explicit attention. **gMLP Block Structure** **Channel Split**: - Input channels are divided into gating branch and value branch. - Each branch receives separate linear transforms. **Spatial Projection**: - Gating branch is projected along token dimension to encode global context. - Projection weights are learned end to end. **Elementwise Gate**: - Value branch is multiplied by projected gate signal. - Output then passes through residual and normalization. **How It Works** **Step 1**: Patch embeddings enter gMLP block, channel split is performed, and gate branch is transformed across tokens. **Step 2**: Gate output modulates value branch by elementwise multiplication, then residual addition and feedforward layers continue. **Tools & Platforms** - **timm**: gMLP variants for rapid benchmarking. - **PyTorch Lightning**: Good for ablation on gate width and depth. - **Inference SDKs**: Gate operations map well to standard tensor kernels. gMLP is **an efficient middle ground between plain MLP mixing and full attention complexity** - its spatial gating unit delivers adaptive context flow with a compact compute profile.