Neural ODEs are deep learning models that define the hidden state dynamics as a continuous ordinary differential equation rather than discrete layers — replacing the sequence of finite transformation layers with a continuous-time flow $dh/dt = f_\theta(h(t), t)$ solved by numerical ODE integrators, enabling adaptive computation depth, memory-efficient training, and principled modeling of continuous-time processes.

From ResNets to Neural ODEs

• ResNet: $h_{t+1} = h_t + f_\theta(h_t)$ — discrete step, fixed number of layers.
• Neural ODE: $\frac{dh}{dt} = f_\theta(h(t), t)$ — continuous transformation, solved from t=0 to t=1.
• ResNet layers are Euler discretizations of the underlying ODE.
• Neural ODE makes this connection explicit → can use sophisticated ODE solvers.

Forward Pass

1. Start with initial condition h(0) = input features. 2. Define dynamics function: $f_\theta(h, t)$ — a neural network. 3. Solve ODE from t=0 to t=T using numerical solver: h(T) = ODESolve(f_θ, h(0), 0, T). 4. h(T) is the output representation.

Backward Pass (Adjoint Method)

• Naive approach: Backprop through ODE solver steps → O(L) memory (like a deep ResNet).
• Adjoint method: Solve a second ODE backwards in time to compute gradients.
• Memory cost: O(1) — constant regardless of number of solver steps.
• Trade-off: Recomputes forward trajectory during backward pass → slightly slower but dramatically less memory.

ODE Solvers Used

Adaptive Computation

• Adaptive solvers take more steps where dynamics are complex, fewer where simple.
• Result: Model automatically allocates more computation to harder inputs.
• During inference: "Easy" inputs processed with fewer function evaluations → faster.

Applications

• Time-Series Modeling: Irregularly-sampled data (medical records, sensor logs) — ODE naturally handles variable time gaps.
• Continuous Normalizing Flows: Invertible generative models with exact log-likelihood.
• Physics-Informed ML: Model physical systems (fluid dynamics, molecular dynamics) with neural ODEs that respect continuous dynamics.

Implementation: torchdiffeq

from torchdiffeq import odeint
h_T = odeint(dynamics_func, h_0, t_span, method='dopri5')

Neural ODEs are a foundational bridge between deep learning and dynamical systems theory — their continuous formulation provides principled tools for modeling temporal processes, enabling adaptive computation, and connecting modern machine learning with centuries of mathematical theory about differential equations.