weight normalization

**Weight Normalization** is a **reparameterization technique that decouples the magnitude and direction of weight vectors** — representing each weight vector as $w = g cdot v/||v||$, where $g$ is a learnable scalar (magnitude) and $v/||v||$ is the unit direction. **How Does Weight Normalization Work?** - **Decomposition**: $w = g cdot hat{v}$ where $hat{v} = v / ||v||$. - **Parameters**: Learn $g$ (scalar magnitude) and $v$ (direction vector) instead of $w$ directly. - **Gradient**: Gradients with respect to $v$ are projected orthogonal to $v$, decoupling magnitude from direction updates. - **Paper**: Salimans & Kingma (2016). **Why It Matters** - **Faster Convergence**: Decoupling magnitude and direction improves conditioning of the optimization. - **No Batch Statistics**: Unlike BatchNorm, weight normalization doesn't depend on batch statistics -> works for any batch size. - **Inference**: No difference between training and inference behavior (unlike BatchNorm). **Weight Normalization** is **polar coordinates for neural network weights** — separating how big the weights are from which direction they point for smoother optimization.