adjoint sensitivity method

**Adjoint Sensitivity Method** is the **memory-efficient technique for computing gradients through ODE solvers** — instead of storing all intermediate states (backpropagation), it solves an adjoint ODE backward in time, reducing memory from $O(L)$ (number of steps) to $O(1)$. **How the Adjoint Method Works** - **Forward Pass**: Solve the ODE $dz/dt = f_ heta(z, t)$ from $t_0$ to $t_1$, storing only the final state. - **Adjoint ODE**: Define $a(t) = dL/dz(t)$ (the adjoint). It satisfies $da/dt = -a^T partial f/partial z$. - **Backward Pass**: Solve the adjoint ODE backward from $t_1$ to $t_0$, simultaneously computing parameter gradients. - **Constant Memory**: Only stores the current state and adjoint — no checkpointing needed. **Why It Matters** - **Memory Efficiency**: Enables Neural ODEs with very deep (continuous) dynamics without memory blow-up. - **Scalability**: Train models with millions of time steps that would be impossible with standard backpropagation. - **Trade-Off**: Adjoint method requires solving an additional ODE backward — trades memory for compute. **Adjoint Sensitivity** is **backpropagation without storing intermediates** — solving an ODE backward to compute gradients with constant memory.