data-dependent initialization

**Data-Dependent Initialization** is a **weight initialization approach that uses a batch of real training data to calibrate initial weights** — adjusting weight magnitudes and biases based on the actual statistics of the data flowing through the network, rather than relying on theoretical assumptions. **How Does Data-Dependent Initialization Work?** - **Forward Pass**: Pass a mini-batch of real data through the network at initialization. - **Calibrate**: Adjust each layer's weights so that output activations have unit variance and zero mean. - **Examples**: LSUV, Data-Dependent Init for normalizing flows, Net2Net-style initialization. - **Contrast**: Theoretical methods (Xavier, He) assume specific input distributions and activation functions. **Why It Matters** - **Accuracy**: Accounts for the actual data distribution, not theoretical i.i.d. assumptions. - **Complex Architectures**: Essential for architectures where theoretical initialization is difficult (normalizing flows, GANs). - **Robustness**: More robust across diverse datasets and preprocessing pipelines. **Data-Dependent Initialization** is **calibration at birth** — using real data to fine-tune the starting conditions for optimal signal flow through any architecture.