Neural Ordinary Differential Equations (Neural ODEs) are a class of deep learning models that replace discrete residual layers with continuous-depth transformations defined by ODEs, where the hidden state evolves according to dh/dt = f_θ(h(t), t) and is integrated using adaptive ODE solvers — offering constant memory training (via adjoint method), adaptive computation, and a principled framework for continuous-time dynamics.

From ResNets to Neural ODEs: A residual network computes h_{t+1} = h_t + f_θ(h_t) — an Euler discretization of a continuous ODE dh/dt = f_θ(h,t). Neural ODEs take the continuous limit: instead of fixed discrete layers, the hidden state evolves continuously from time t=0 to t=T, with the ODE solved by a numerical integrator (Dopri5, RK45, or adaptive-step solvers).

Forward Pass: Given input h(0), solve the initial value problem dh/dt = f_θ(h(t), t) from t=0 to t=T using an off-the-shelf ODE solver. The solver adaptively chooses step sizes for accuracy — using more function evaluations in regions where dynamics change rapidly and fewer where they are smooth. This provides adaptive computation — complex inputs automatically receive more computation.

Backward Pass (Adjoint Method): Naive backpropagation through the ODE solver would require storing all intermediate states — O(L) memory where L is the number solver steps. The adjoint method instead: defines the adjoint a(t) = dL/dh(t), derives an adjoint ODE da/dt = -a^T · ∂f/∂h that runs backward in time, and computes parameter gradients by integrating: dL/dθ = -∫ a^T · ∂f/∂θ dt. This requires only O(1) memory (constant regardless of depth/steps), enabling very deep effective networks.

Applications:

Continuous Normalizing Flows (CNFs): A key application. Standard normalizing flows use discrete bijective transformations with restricted architectures (to ensure invertibility). CNFs use the instantaneous change of variables formula: d(log p)/dt = -tr(∂f/∂h), which places no restrictions on f_θ — any neural network can define the dynamics. The Hutchinson trace estimator approximates tr(∂f/∂h) stochastically, making this practical for high dimensions.

Limitations: Training speed — ODE solvers are inherently sequential (each step depends on the previous), making Neural ODEs slower to train than discrete networks; stiffness — some learned dynamics become stiff (requiring many tiny steps), increasing computation; expressiveness — single-trajectory ODEs cannot represent certain transformations (crossing trajectories are forbidden by uniqueness theorems); and hyperparameter sensitivity — solver tolerance affects both accuracy and speed.

Neural ODEs opened a new paradigm connecting deep learning with dynamical systems theory — demonstrating that the tools of differential equations, numerical analysis, and continuous mathematics have deep correspondences with neural network architectures, inspiring a rich research direction in scientific machine learning.