Skip Connection

A skip connection is any link in a neural network that carries a layer's output past one or more intermediate layers and merges it into a later layer, so that information and gradients have a short path alongside the long one. It is the general family of which the residual connection (covered in its own entry) is the best-known member; the family also includes gated shortcuts, concatenative dense links, and the long bridges between the two halves of an encoder-decoder. What unites them is the design stance that a deep network should not force every signal to pass through every transformation.

The family tree begins with highway networks and ResNet, which differ in whether the shortcut is gated. Srivastava, Greff, and Schmidhuber's 2015 "Highway Networks" borrowed the gating idea from LSTMs: a learned transform gate decides, per unit, how much of the layer's computed output and how much of its untouched input to pass on, and this allowed networks of hundreds of layers to be trained with plain gradient descent. He, Zhang, Ren, and Sun's "Deep Residual Learning for Image Recognition" (2015, CVPR 2016) removed the gate and used a pure identity shortcut added to the block's output, a simplification so effective that a 152-layer ResNet won the ILSVRC 2015 classification task. The addition y = x + F(x) needs the two sides to have the same shape, so when a block changes the channel count or resolution, the shortcut uses a 1×1 convolution as a projection.

DenseNet changes the merge operation from addition to concatenation, and with it the economics of the network. In Huang, Liu, van der Maaten, and Weinberger's DenseNet (CVPR 2017, best paper award), every layer inside a dense block receives the stacked feature maps of all earlier layers and passes its own output to all later ones. Because features are appended, not summed, earlier information stays intact and separable, and each layer can be very thin, adding only a small growth rate of new channels; the price is that memory use grows quickly, since the concatenated input to later layers keeps widening. The distinction matters in practice: addition mixes the old and new signal into one tensor, whereas concatenation lets the next layer decide how to weight each source through its own learned weights.

U-Net popularized the long skip, which serves a different purpose than the short ones: preserving spatial detail rather than easing optimization. Ronneberger, Fischer, and Brox's 2015 biomedical segmentation network contracts an image through an encoder to capture context and then expands it through a decoder to recover resolution, and at each resolution level it concatenates the encoder's feature maps onto the decoder's. Without those bridges the decoder would have to reconstruct fine boundaries from a coarse bottleneck alone; with them, the precise location information from early layers arrives directly. Fully convolutional networks (Long, Shelhamer, and Darrell, 2015) had used summed skips for the same reason, and the pattern now underlies the denoising networks inside many image diffusion models, as well as feature pyramids in object detection.

Why skips help is still described in several complementary ways, and none of them is the whole story. The gradient view says the shortcut gives backpropagation an unobstructed route to early layers, countering vanishing gradients; the optimization-landscape view, supported by Li, Xu, Taylor, Studer, and Goldstein's 2018 loss visualizations, finds that skips make the surface smoother and easier to descend; and Veit, Wilber, and Belongie's 2016 analysis found that residual networks behave like ensembles of many relatively shallow paths. The transformer inherits the idea wholesale, with a shortcut around every attention and feed-forward sublayer, which is part of why stacks of dozens or hundreds of blocks train at all.

Common misconceptions are that skip connection and residual connection are synonyms, that more skips are always better, and that the shortcut is free. Residual connections are the additive subset; a concatenative or long skip is a different mechanism. Dense wiring multiplies memory traffic, a long skip in a segmentation network can leak low-level artifacts or shortcut-learn trivial copies if the bottleneck is not narrow enough, and a mismatched shape or a badly placed normalization layer can quietly erase the benefit. Whether a shortcut helps is therefore an architectural choice about what should be preserved, and for what purpose.

Read skip connection as a deliberate bypass lane in a deep network: its job is to keep earlier information and gradient flow alive, and the merge rule (add, gate, or concatenate) and the length of the jump decide whether it is eased optimization, feature reuse, or recovered fine detail.