residual connection

**Residual Connections (Skip Connections)** — shortcut paths that add a layer's input directly to its output: $y = F(x) + x$, enabling training of very deep networks. **Problem Solved** - Deep networks (50+ layers) suffered degradation: adding more layers made accuracy WORSE - Not overfitting — even training accuracy degraded - Deeper networks couldn't learn identity mappings through many nonlinear layers **How Residuals Help** - Network only needs to learn the residual $F(x) = H(x) - x$ (the difference from identity) - If a layer isn't useful, weights can go to zero and the signal passes through unchanged - Gradients flow directly through skip connections — no vanishing **ResNet Results (2015)** - Won ImageNet with 152 layers (previously ~20 was practical) - Enabled training networks with 1000+ layers - Became standard in virtually all deep architectures **Beyond Vision** - Transformers use residual connections around every attention and FFN block - Key enabler for training deep language models (GPT, BERT) **Residual connections** are arguably the single most important architectural innovation in deep learning after backpropagation itself.