zero-init residual

**Zero-Init Residual** is an **initialization strategy where the last layer (e.g., final BN or convolution) of each residual branch is initialized to zero** — making each residual block output the identity function at initialization, similar in spirit to SkipInit. **How Does Zero-Init Residual Work?** - **Standard**: Initialize all layers' weights with standard initialization (He, Xavier). - **Zero-Init**: Set the weight (or BN $gamma$) of the last layer in each residual block to 0. - **Effect**: At initialization, $F(x) = 0$ so $y = x + F(x) = x$ (identity). - **Used In**: ResNet (zero-init last BN $gamma$), GPT-2 (zero-init last linear in each block). **Why It Matters** - **Training Stability**: Enables stable training of very deep networks from the start. - **Standard Practice**: Used by default in modern ResNet implementations and GPT-style transformers. - **Gradient Flow**: Identity initialization ensures gradients flow directly through skip connections at the start. **Zero-Init Residual** is **the default start for residual networks** — making each block initially transparent so gradients flow freely from the very first step.