gradient centralization

**Gradient Centralization (GC)** is a **simple optimization technique that centralizes (zero-means) gradients before each update** — subtracting the mean of the gradient vector from each element, which acts as a regularizer and improves training stability and generalization. **How Does Gradient Centralization Work?** - **Operation**: For each weight tensor, compute $hat{g} = g - ar{g}$ where $ar{g}$ is the column-wise mean. - **Constraint**: The resulting update has zero mean -> constrains the weight space. - **Integration**: Applied as a single line of code before the optimizer update step. - **Paper**: Yong et al. (2020). **Why It Matters** - **Simplicity**: One line of code, no additional hyperparameters, works with any optimizer. - **Regularization**: Acts as implicit regularization by constraining the update direction. - **Performance**: Consistently improves both convergence speed and final accuracy by 0.1-0.5%. **Gradient Centralization** is **the zero-mean trick for gradients** — a remarkably simple technique that improves training for free.