switchable normalization

**Switchable Normalization** is a **meta-normalization technique that learns to combine BatchNorm, InstanceNorm, and LayerNorm** — using learnable weights to adaptively select the optimal normalization method for each layer and each channel during training. **How Does Switchable Normalization Work?** - **Three Statistics**: Compute BN, IN, and LN statistics simultaneously. - **Learnable Weights**: $hat{mu} = lambda_{BN}mu_{BN} + lambda_{IN}mu_{IN} + lambda_{LN}mu_{LN}$ (and same for variance). - **Softmax**: Weights are softmax-normalized -> always sum to 1. - **Learning**: The network learns which normalization is best for each layer. - **Paper**: Luo et al. (2019). **Why It Matters** - **Automatic Selection**: No need to manually choose between BN, IN, LN — the network decides. - **Task-Adaptive**: Different tasks (classification, style transfer, detection) benefit from different normalizations. - **Insight**: Analysis of learned weights reveals which normalization is preferred at different depths and for different tasks. **Switchable Normalization** is **letting the network choose its own normalization** — a meta-learning approach that adapts normalization strategy per layer.