scaled initialization

**Scaled Initialization** is a **weight initialization strategy that scales initial weights based on network depth** — ensuring that the variance of activations and gradients remains stable as they propagate through many layers, preventing signal explosion or vanishing. **How Does Scaled Initialization Work?** - **Principle**: Scale weights by $1/sqrt{n}$ where $n$ depends on the method (fan-in, fan-out, or both). - **Depth Scaling**: For very deep networks, additionally scale by $1/sqrt{L}$ or $1/sqrt{2L}$ where $L$ is the number of layers. - **Examples**: GPT-2 scales residual projections by $1/sqrt{N}$ where $N$ is the number of layers. - **Goal**: Maintain unit variance of activations and gradients throughout the entire network at initialization. **Why It Matters** - **Deep Networks**: Without proper scaling, signals either explode or vanish after many layers. - **Foundation Models**: Scaled initialization is critical for training 100+ layer transformers. - **Theory**: Connects to mean field theory — maintaining criticality at initialization. **Scaled Initialization** is **the volume control for deep networks** — setting initial weights at exactly the right level so signals propagate cleanly through every layer.