resmlp for vision

**ResMLP** is the **residual all-MLP architecture that simplifies Mixer style blocks with affine normalization and strong skip design for stable optimization** - it aims for better data efficiency and training behavior while preserving the attention-free philosophy. **What Is ResMLP?** - **Definition**: An MLP based vision model that combines token interaction layers, channel MLPs, and residual blocks with lightweight normalization. - **Normalization Choice**: Uses affine transforms instead of full LayerNorm in core blocks. - **Residual Emphasis**: Strong identity paths keep gradients stable through deep stacks. - **Training Recipe**: Heavy augmentation and regularization are important for top performance. **Why ResMLP Matters** - **Optimization Stability**: Residual plus affine design can converge more reliably in deep all-MLP setups. - **Data Efficiency**: Often performs better than earlier Mixer variants on moderate scale datasets. - **Low Complexity**: Keeps operator set small for easier deployment and profiling. - **Interpretability**: Learned token-mixing weights often resemble structured spatial filters. - **Architecture Insight**: Shows that normalization and residual details are as important as block type. **ResMLP Components** **Token Interaction Layer**: - Mixes patch tokens with learned linear transforms. - Works globally across the patch sequence. **Channel Feedforward Layer**: - Expands channel dimension, applies nonlinearity, then projects back. - Supplies semantic capacity per token. **Affine Residual Wrapper**: - Applies trainable scale and shift around residual paths. - Stabilizes updates at initialization. **How It Works** **Step 1**: Patchify image, project to embeddings, and run token interaction with residual addition to distribute spatial context. **Step 2**: Run channel feedforward with affine scaling, repeat across stages, then pool and classify. **Tools & Platforms** - **timm**: Provides ResMLP variants and training scripts. - **PyTorch**: Easy to customize affine and residual parameters for experiments. - **WandB**: Useful for tracking sensitivity to normalization and depth. ResMLP is **a practical evolution of all-MLP vision design that trades unnecessary complexity for cleaner residual dynamics** - it helps teams reach strong results with a compact and understandable architecture.