Partial Derivative

A partial derivative is the rate at which a function of several variables changes when exactly one of those variables is nudged and all the others are held fixed. For a function f(x, y), the partial derivative with respect to x, written ∂f/∂x, treats y as if it were a constant and differentiates in x as in ordinary single-variable calculus. It answers a narrow, local question: standing at this point, how steeply does the output rise if only this one input moves? Stacking the answers for every input produces the gradient, and that stack is the central object of nearly all of modern machine learning optimization.

Computing a partial derivative needs no new rules, only a discipline about what counts as a constant. Take f(x, y) = x²y + 3y. To find ∂f/∂x, freeze y: the term x²y becomes y times x², whose derivative is 2xy, and the term 3y is a constant and vanishes, so ∂f/∂x = 2xy. To find ∂f/∂y, freeze x: x²y has derivative x² and 3y has derivative 3, so ∂f/∂y = x² + 3. At the point (2, 1) the two partials are 4 and 7, meaning a small step along x raises f about four times the step size while the same step along y raises it about seven times. The curved symbol ∂, a stylized d popularized by Carl Gustav Jacobi in the nineteenth century, exists precisely to warn that other variables are being held still.

The gradient collects all the partial derivatives into one vector, and that vector points in the direction of steepest local increase. Its length is the steepness in that direction, and moving the opposite way gives the steepest local decrease, which is why gradient descent subtracts it. A neural network with a billion weights has a loss function of a billion variables, and its gradient is simply a list of a billion partial derivatives, one per weight, each saying how the loss would change if that single weight alone were adjusted. Higher-order objects follow the same pattern: the Jacobian arranges the partial derivatives of a vector-valued function into a matrix, and the Hessian arranges all second partial derivatives of a scalar function, encoding curvature.

Mixed partial derivatives usually commute, but the guarantee has conditions. Differentiating first in x and then in y gives the same result as the reverse order whenever the second partials are continuous, a result associated with Clairaut and Schwarz. That symmetry is why the Hessian of a smooth loss is a symmetric matrix, a fact that optimization theory relies on constantly. Pathological functions exist where the two orders disagree, but they rarely appear in practice.

Existing partial derivatives do not guarantee that a function is smoothly differentiable, which is the most common conceptual trap. A function can have both partials at a point and still fail to be continuous there, because partials only probe movement along the coordinate axes and say nothing about diagonal approaches. Genuine differentiability requires a good linear approximation in every direction, and a sufficient condition is that the partials exist and are continuous nearby. A second pitfall is notational: writing ∂f/∂x is meaningless without saying which other variables are held fixed, which matters in thermodynamics and in any setting where variables are coupled, since the derivative of f with respect to x changes depending on whether y or some function of x and y is held constant.

In practice partial derivatives are rarely worked out by hand and are rarely approximated by finite differences either. Nudging each of a billion weights separately to measure its effect would need a billion forward passes, so training relies on the chain rule applied in reverse, which delivers every partial derivative of the loss in roughly the cost of a couple of forward passes. Finite differences survive mainly as a gradient check: compute one partial numerically as (f(x + h) − f(x − h)) / 2h for a small h and compare it with the automatic result to catch implementation bugs. Libraries such as PyTorch, TensorFlow and JAX expose these values directly, and in JAX the gradient of a function is a single call.

Read partial derivative as one component of a slope: it measures how a quantity responds to a single input while everything else stays put, and gathering all such components into the gradient is what turns a landscape of many variables into a direction that an optimizer can follow downhill.