ode-rnn
**ODE-RNN** is a **hybrid sequence model that combines Neural ODEs for continuous-time state evolution between observations with Recurrent Neural Networks for discrete state updates at observation times** — addressing the irregular time series challenge by modeling the continuous dynamics of a hidden state between measurement events and incorporating each new observation via a standard gated RNN update, providing a practical middle ground between purely continuous Neural ODE models and discrete RNNs that lack principled continuous-time semantics.
**Motivation: The Best of Both Worlds**
Standard RNNs process sequences at discrete time steps: h_{n+1} = RNN(h_n, x_{n+1}). For irregular sequences, this creates two problems:
1. The model cannot distinguish Δt = 1 hour from Δt = 1 day — both produce the same update
2. Zero-padding for missing time steps introduces artificial "no observation" signals that bias the hidden state
Neural ODEs provide continuous-time dynamics but are purely deterministic between observations — they cannot incorporate new information from sparse observations without adding encoder complexity (as in Latent ODEs).
ODE-RNN solves this by splitting the processing into two distinct phases:
**Phase 1 — Between observations (Neural ODE)**: Given current hidden state h(tₙ) and next observation time tₙ₊₁, integrate the ODE:
h(tₙ₊₁⁻) = h(tₙ) + ∫_{tₙ}^{tₙ₊₁} f(h(s), s; θ_ode) ds
The state evolves continuously, with dynamics that decay or oscillate according to the learned vector field f.
**Phase 2 — At observations (GRU/LSTM update)**: Incorporate the new observation xₙ₊₁ using a standard gated RNN:
h(tₙ₊₁) = GRU(h(tₙ₊₁⁻), xₙ₊₁)
The RNN update can also be replaced by an attention mechanism for long-range dependencies.
**Architecture Diagram**
h(t₀) →[Neural ODE: t₀→t₁]→ h(t₁⁻) →[GRU+x₁]→ h(t₁) →[Neural ODE: t₁→t₂]→ h(t₂⁻) →[GRU+x₂]→ h(t₂) → ...
The Neural ODE segments can have arbitrary, different durations — Δt₁ ≠ Δt₂ — and the model correctly accounts for this through the integration.
**Temporal Decay Properties**
The Neural ODE dynamics between observations can implement several principled behaviors:
- **Exponential decay**: f(h) = -λh forces the state to decay toward zero between observations (appropriate for sensor readings that become stale)
- **Oscillatory dynamics**: f(h) = Ah (linear system) captures periodic patterns in the underlying process
- **Arbitrary nonlinear dynamics**: The full neural network f(h, t; θ) can represent complex attractor dynamics
For many real-world processes, the learned dynamics often resemble exponential decay — the model effectively learns to discount stale information.
**Comparison to Alternative Models**
| Model | Irregular Handling | Uncertainty | Complexity | Best For |
|-------|-------------------|-------------|------------|---------|
| **Standard RNN** | Poor (fixed Δt assumed) | None | Low | Regular sequences |
| **GRU-D** | Time decay heuristic | None | Low | Simple irregular series |
| **ODE-RNN** | Principled ODE | Low (deterministic) | Medium | Prediction, classification |
| **Latent ODE** | Principled ODE | High (probabilistic) | High | Generation, imputation |
| **Neural CDE** | Controlled path | Medium | Medium | Control tasks |
**Applications**
**Electronic Health Records**: Clinical notes, lab values, and vital signs arrive at irregular intervals determined by patient condition and care protocols. ODE-RNN outperforms standard LSTM on mortality prediction and disease onset prediction by properly accounting for time elapsed between measurements.
**Event-Based Sensors**: Neuromorphic cameras and event-based IMUs generate observations asynchronously. ODE-RNN processes these sparse event streams without discretization artifacts.
**Financial Market Data**: High-frequency trading data has variable inter-trade intervals. ODE-RNN captures the continuous price dynamics between trades rather than artificially resampling to a fixed grid.
ODE-RNN is implemented in the torchdiffeq library (alongside Neural ODEs) and has been replicated in Julia's DifferentialEquations.jl ecosystem. The simple conceptual structure — ODE between observations, RNN at observations — makes it the most accessible entry point to continuous-time sequence modeling.