lsuv

**LSUV** (Layer-Sequential Unit-Variance) is a **data-driven initialization method that iteratively adjusts each layer's weights to produce unit-variance activations** — using a mini-batch of real data to empirically calibrate the initialization, accounting for non-linearities and architectural specifics. **How Does LSUV Work?** 1. **Initialize**: Start with orthogonal initialization. 2. **Forward Pass**: Pass a mini-batch through the network. 3. **Per Layer**: Measure the variance of each layer's activations. 4. **Rescale**: Multiply weights by $1/sqrt{ ext{Var}(output)}$ to achieve unit variance. 5. **Iterate**: Repeat until all layers have unit-variance activations. **Why It Matters** - **Data-Driven**: Accounts for the actual data distribution, not just theoretical assumptions. - **Architecture-Agnostic**: Works for any architecture (CNNs, RNNs, exotic activations). - **Post-Init Calibration**: Can be applied after any initialization to fix variance issues. **LSUV** is **empirical initialization calibration** — using real data to tune each layer's scale for perfect signal propagation, regardless of the theoretical assumptions.