neural ode
**Neural ODEs** are **deep learning models that define the hidden state dynamics as a continuous ordinary differential equation rather than discrete layers** — replacing the sequence of finite transformation layers with a continuous-time flow $dh/dt = f_\theta(h(t), t)$ solved by numerical ODE integrators, enabling adaptive computation depth, memory-efficient training, and principled modeling of continuous-time processes.
**From ResNets to Neural ODEs**
- **ResNet**: $h_{t+1} = h_t + f_\theta(h_t)$ — discrete step, fixed number of layers.
- **Neural ODE**: $\frac{dh}{dt} = f_\theta(h(t), t)$ — continuous transformation, solved from t=0 to t=1.
- ResNet layers are Euler discretizations of the underlying ODE.
- Neural ODE makes this connection explicit → can use sophisticated ODE solvers.
**Forward Pass**
1. Start with initial condition h(0) = input features.
2. Define dynamics function: $f_\theta(h, t)$ — a neural network.
3. Solve ODE from t=0 to t=T using numerical solver: `h(T) = ODESolve(f_θ, h(0), 0, T)`.
4. h(T) is the output representation.
**Backward Pass (Adjoint Method)**
- Naive approach: Backprop through ODE solver steps → O(L) memory (like a deep ResNet).
- **Adjoint method**: Solve a second ODE backwards in time to compute gradients.
- Memory cost: O(1) — constant regardless of number of solver steps.
- Trade-off: Recomputes forward trajectory during backward pass → slightly slower but dramatically less memory.
**ODE Solvers Used**
| Solver | Order | Steps | Adaptive | Use Case |
|--------|-------|-------|----------|----------|
| Euler | 1 | Fixed | No | Fast, low accuracy |
| RK4 (Runge-Kutta) | 4 | Fixed | No | Good accuracy |
| Dopri5 (RK45) | 5(4) | Adaptive | Yes | Default choice |
| Adams (multistep) | Variable | Adaptive | Yes | Stiff systems |
**Adaptive Computation**
- Adaptive solvers take more steps where dynamics are complex, fewer where simple.
- Result: Model automatically allocates more computation to harder inputs.
- During inference: "Easy" inputs processed with fewer function evaluations → faster.
**Applications**
- **Time-Series Modeling**: Irregularly-sampled data (medical records, sensor logs) — ODE naturally handles variable time gaps.
- **Continuous Normalizing Flows**: Invertible generative models with exact log-likelihood.
- **Physics-Informed ML**: Model physical systems (fluid dynamics, molecular dynamics) with neural ODEs that respect continuous dynamics.
**Implementation: torchdiffeq**
```
from torchdiffeq import odeint
h_T = odeint(dynamics_func, h_0, t_span, method='dopri5')
```
Neural ODEs are **a foundational bridge between deep learning and dynamical systems theory** — their continuous formulation provides principled tools for modeling temporal processes, enabling adaptive computation, and connecting modern machine learning with centuries of mathematical theory about differential equations.