Neural ODEs (Neural Ordinary Differential Equations) define neural network layers as continuous-depth transformations governed by ordinary differential equations — where the hidden state evolves according to $dh/dt = f(h, t; heta)$ and the forward pass is computed by integrating this ODE from $t=0$ to $t=1$ — bridging deep learning and dynamical systems theory to enable adaptive computation depth, constant memory training via the adjoint method, and natural modeling of continuous-time processes like physics simulations and irregular time series.

What Are Neural ODEs?

• Standard ResNet: $h_{t+1} = h_t + f(h_t, heta_t)$ — discrete steps with fixed depth.
• Neural ODE: $dh/dt = f(h, t; heta)$ — continuous transformation where the network "depth" is the integration time.
• Forward Pass: Use an ODE solver (Runge-Kutta, Dormand-Prince) to integrate from initial state to final state.
• Backward Pass: The adjoint method computes gradients without storing intermediate states — $O(1)$ memory regardless of integration steps.
• Key Paper: Chen et al. (NeurIPS 2018), "Neural Ordinary Differential Equations" — Best Paper Award.

Why Neural ODEs Matter

• Memory Efficiency: The adjoint method computes exact gradients with constant memory, unlike backpropagation through discrete layers which requires $O(L)$ memory for $L$ layers.
• Adaptive Computation: The ODE solver automatically uses more function evaluations for complex inputs and fewer for simple ones — the network "depth" adapts to input difficulty.
• Continuous Dynamics: Natural framework for modeling physical systems, chemical reactions, population dynamics, and any process described by differential equations.
• Irregular Time Series: Unlike RNNs (which require regular time steps), neural ODEs handle irregularly sampled observations natively by integrating between observation times.
• Invertibility: Neural ODEs define invertible transformations, enabling continuous normalizing flows (FFJORD) with free-form Jacobians.

Architecture and Training

Neural ODE Variants

• Neural SDEs: Add stochastic noise $dh = f(h,t; heta)dt + g(h,t; heta)dW$ for uncertainty quantification and generative modeling.
• Augmented Neural ODEs: Expand state dimension to overcome topological limitations of standard neural ODEs.
• FFJORD: Continuous normalizing flows using neural ODEs — free-form Jacobian enables more expressive density estimation than coupling flows.
• Latent ODEs: Encode irregular time series into latent initial conditions, then integrate a neural ODE forward for prediction.
• Neural CDEs (Controlled DEs): Extend neural ODEs to handle streaming input data, bridging neural ODEs and RNNs.

Applications

• Physics-Informed ML: Model physical systems where governing equations are partially known — combine neural ODEs with domain knowledge.
• Irregular Time Series: Clinical data (vital signs at irregular intervals), financial data (tick-by-tick trades), and sensor data with missing measurements.
• Generative Modeling: FFJORD provides continuous normalizing flows with exact likelihoods and efficient sampling.
• Robotics: Model continuous dynamics of robotic systems for control and planning.

Neural ODEs are the unification of deep learning and dynamical systems — proving that neural networks and differential equations are two perspectives on the same mathematical object, and opening a rich design space where centuries of ODE theory meets modern deep learning.