Chain Rule
The chain rule is the calculus rule for differentiating a composition of functions: if y depends on u and u depends on x, then the rate of change of y with respect to x is the rate of change of y with respect to u multiplied by the rate of change of u with respect to x. Written compactly, d/dx f(g(x)) = f'(g(x)) × g'(x). A deep neural network is nothing but a long composition of simple functions, so the chain rule is the exact identity that lets the influence of every parameter on the final loss be computed layer by layer, and applying it in a particular order is the algorithm known as backpropagation.
Take a one-neuron example to see the mechanics. Let z = w×x + b, a = σ(z) with σ the sigmoid, and the loss L = (a − t)² for a target t. The derivative of the loss with respect to the weight w factors into three links: ∂L/∂a = 2(a − t), then ∂a/∂z = a(1 − a), then ∂z/∂w = x. Multiplying them gives ∂L/∂w = 2(a − t) × a(1 − a) × x. Each factor is a local derivative that depends only on one operation and the values stored during the forward pass, and the product threads the error signal from the output back to the parameter. For vector-valued layers the same product becomes a chain of Jacobian matrices.
Backpropagation is the chain rule evaluated in reverse order over a recorded computation graph, which is what makes it cheap. Multiplying the chain from the output end means each intermediate quantity is a vector of sensitivities of the single scalar loss, so one backward sweep yields the gradient with respect to all parameters at once, at a cost typically a small constant multiple of the forward pass. Evaluating from the input end instead, called forward mode, would need one sweep per input and is efficient only when there are few inputs and many outputs. Deep learning has millions to trillions of inputs and one scalar output, so reverse mode wins decisively. The price is memory, because the forward activations must be kept until the backward pass needs them, which motivates activation checkpointing.
The history is more layered than the single famous citation suggests. Reverse-mode differentiation was described by Seppo Linnainmaa in his 1970 master's thesis on rounding error, and Paul Werbos proposed applying it to training neural networks in his 1974 doctoral thesis. The paper that made it widely known in the field was Rumelhart, Hinton and Williams, "Learning representations by back-propagating errors", published in Nature in 1986, which showed that hidden layers could learn useful internal representations. Credit is therefore split between the mathematics, earlier proposals, and the 1986 demonstration that turned it into a practical research program.
Automatic differentiation is the general machinery behind this, and it is neither symbolic nor numerical differentiation. Symbolic differentiation manipulates formulas and can explode in size; numerical differentiation uses finite differences and suffers from truncation and rounding error. Automatic differentiation records the elementary operations executed and applies the chain rule to those exact steps, giving derivatives accurate to machine precision. PyTorch builds the graph dynamically during execution, TensorFlow originally defined it statically, and JAX transforms Python functions through composable tools such as grad. Because the engine differentiates what the program actually ran, ordinary control flow such as loops and conditionals is handled naturally.
Several recurring misconceptions are worth correcting. Backpropagation is not a learning algorithm by itself: it only computes gradients, and an optimizer such as stochastic gradient descent decides how to use them. It is not an exact model of how biological brains learn, and the question of whether cortical circuits approximate anything like it remains debated. And because gradients are products of many local factors, long chains can shrink them toward zero or blow them up, giving the vanishing and exploding gradient problems that residual connections, normalization, careful initialization and gradient clipping were developed to tame.
Read chain rule as the bookkeeping law of influence through a pipeline: each stage multiplies in its own local sensitivity, and running those multiplications backward from the loss, as backpropagation does, delivers the effect of every parameter on the result in a single sweep.