skipinit

**SkipInit** is an **initialization technique for residual networks that multiplies each residual path by a learnable scalar initialized to zero** — ensuring that at initialization, the network is equivalent to a shallow network (identity function), enabling training of extremely deep networks without BatchNorm. **How Does SkipInit Work?** - **Standard Residual**: $y = x + F(x)$ - **SkipInit**: $y = x + alpha cdot F(x)$ where $alpha$ is initialized to 0. - **At Init**: $y = x$ (identity mapping). The network is effectively 1 layer deep. - **During Training**: $alpha$ grows from 0, gradually introducing the residual contributions. - **Paper**: De & Smith (2020). **Why It Matters** - **No BatchNorm Needed**: Enables training 10,000+ layer ResNets without any normalization. - **Simplicity**: One scalar parameter per residual block. Trivial to implement. - **Theory**: Connects to the insight that deep networks train best when they start as shallow networks and gradually deepen. **SkipInit** is **starting as nothing** — initializing each residual pathway to zero so the model begins as a simple identity and gradually builds complexity.