Calculus for Machine Learning
Calculus for machine learning is the small part of differential calculus that turns learning into an optimization problem: derivatives measure how quickly a quantity changes, gradients collect those rates for many parameters at once, and following them downhill is how a model improves. The derivative of a function at a point is the slope of the curve there, the instantaneous rate of change of the output per unit change of the input. A model with millions or billions of adjustable weights defines a loss, a single number measuring its error, and calculus answers the only question training needs: for each weight, if it is nudged up slightly, does the loss go up or down, and by how much?
The gradient is the vector of the loss's partial derivatives, one for each parameter, and it points in the direction of steepest increase. Moving against it lowers the loss fastest for a small step. Gradient descent, an idea traced to Augustin-Louis Cauchy in 1847, repeats a simple update: new weight equals old weight minus the learning rate times the gradient. Take the loss (w − 3) squared, whose derivative is 2 times (w − 3), start at w = 0 with a learning rate of 0.1: the gradient is −6, so w moves to 0.6; there the gradient is −4.8, so w moves to 1.08, and the steps shrink as the minimum at 3 is approached. The learning rate is critical: too small and training crawls, too large and the updates overshoot and diverge. The separate entries on partial derivatives and the chain rule treat the two tools used to compute gradients.
Backpropagation and automatic differentiation make gradients affordable at scale. A network is a long composition of simple functions, so the derivative of the loss with respect to early weights follows from multiplying local derivatives layer by layer, working backward from the output. The 1986 paper by Rumelhart, Hinton and Williams popularized this for neural networks, and modern frameworks such as PyTorch, TensorFlow and JAX implement the general technique, automatic differentiation, recording the operations of a computation and replaying them in reverse. This differs from numerical differentiation, which approximates derivatives by tiny finite differences and is slow and imprecise, and from symbolic differentiation, which manipulates formulas and can blow up in size. One backward pass costs a small constant multiple of a forward pass regardless of the number of weights, which is what makes training billion-parameter models feasible.
Stochastic and adaptive optimizers refine plain gradient descent. Computing the gradient over the whole dataset is expensive, so stochastic gradient descent, rooted in Robbins and Monro's 1951 stochastic approximation, estimates it from small random batches, trading a noisier direction for far cheaper steps. Momentum smooths the noise across steps, and Adam, introduced by Kingma and Ba in 2014, keeps running averages of the gradient and its square to give each parameter its own effective step size. Second derivatives, gathered in the Hessian matrix, describe curvature and underlie Newton-type methods and analyses of step sizes, but the Hessian of a large network is far too big to compute directly, so practical training relies on first-order information. Integrals appear elsewhere, in the expectations and normalizing constants of probability.
Common misconceptions concern what the gradient is and what it can do. Gradients are taken with respect to the parameters, treating the training data as fixed, not with respect to the data. Deep loss surfaces are not mainly obstructed by bad local minima: Dauphin and colleagues' 2014 work argued that saddle points, where the gradient is zero but the surface curves up in some directions and down in others, are the more common obstacle in high dimensions. A zero gradient does not guarantee a minimum, and a non-differentiable point such as the corner of ReLU at zero is handled by a convention that picks a valid slope. Vanishing and exploding gradients arise because backpropagation multiplies many factors, so values below or above one compound across depth. Finally, descent only finds a minimum of the loss that was written down, so a loss that does not reflect the real goal gets optimized faithfully anyway.
Read calculus for machine learning as the machinery that tells a model which way is downhill: derivatives give the local slope, the gradient collects the slopes for every parameter, and backpropagation plus gradient descent turn that information into the repeated small corrections that constitute training.