resnet

ResNet (Residual Network) introduced skip connections that add the input of a layer to its output, enabling training of very deep networks by addressing the vanishing gradient problem. The key innovation is the residual block: y = F(x) + x, where F(x) is the learned transformation and x is the identity shortcut. This reformulation makes it easier to learn identity mappings—if the optimal transformation is close to identity, the network only needs to learn small residuals. Skip connections provide gradient highways that allow gradients to flow directly through the network during backpropagation, preventing vanishing gradients in deep networks. ResNet demonstrated that deeper networks (152+ layers) outperform shallower ones when skip connections are used, contradicting earlier findings that very deep networks degrade. ResNet variants include ResNeXt (grouped convolutions), Wide ResNet (wider layers), and ResNeSt (split-attention). Skip connections have become a foundational component in modern architectures including transformers (residual connections around attention and feedforward layers). ResNet represents a breakthrough that enabled the deep learning revolution.