mean field theory

**Mean Field Theory** applied to deep learning is a **mathematical framework that analyzes how signals propagate through randomly initialized neural networks** — determining the conditions under which forward signals and backward gradients neither explode nor vanish. **What Is Mean Field Theory for DNNs?** - **Approach**: Treat each neuron's pre-activation as a random variable. Compute the mean and variance of activations layer by layer. - **Order Parameters**: Track the mean ($q^*$) and variance ($chi^*$) of activations across layers. - **Critical Point**: At the "edge of chaos," signals propagate stably — neither growing nor shrinking. - **Initialization**: Use mean field theory to derive optimal weight initialization (e.g., He, Xavier). **Why It Matters** - **Trainability**: Networks initialized at the critical point train fastest (gradients propagate well). - **Depth**: Explains why very deep networks are hard to train without careful initialization. - **Batch Normalization**: Mean field theory explains why BatchNorm works — it dynamically maintains criticality. **Mean Field Theory** is **the physics of signal propagation through deep networks** — determining whether information flows or dies as it passes through millions of parameters.