residual

Residual connections (skip connections) add the input of a layer directly to its output, enabling gradient flow through deep networks by providing an identity shortcut that makes learning identity functions trivial. Origin: introduced in ResNet (2015) to enable training of very deep networks (100+ layers) that were previously impossible to optimize. Formulation: y = F(x) + x, where F(x) is the learned transformation; if F learns zero, the layer computes identity. Gradient flow: gradients flow directly through addition operation; avoids vanishing gradients through many layers. Pre-activation: variant where normalization and activation come before convolution; mathematically cleaner gradient paths. Bottleneck blocks: 1×1 → 3×3 → 1×1 convolutions for efficiency in very deep networks. Dimension matching: when dimensions change, use 1×1 convolution or pooling on skip path. Transformer application: residual around each attention and FFN block; crucial for training deep transformers. Highway networks: predecessor with learned gating; residual connections are ungated (simpler, works better). Dense connections: DenseNet concatenates all previous features instead of adding; different trade-offs. Initialization: with residual connections, layers can start near identity; gradual learning of perturbations. Universal presence: residual connections now standard in virtually all deep architectures. Residual connections are foundational architectural innovation enabling modern deep learning.