Neural Ordinary Differential Equations

The paper introduces a new family of deep neural network models in which, rather than specifying a discrete sequence of hidden layers, the derivative of the hidden state is parameterized by a neural network and the output is computed with a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed, and the authors show how to scalably backpropagate through any ODE solver without access to its internal operations, allowing end-to-end training within larger models.

The authors demonstrate these properties in continuous-depth residual networks and continuous-time latent-variable models, and they construct continuous normalizing flows, a generative model that can be trained by maximum likelihood without partitioning or ordering the data dimensions. By recasting network depth as a continuous quantity governed by numerical ODE solvers, the work provides a memory-efficient and flexible alternative to conventional layered architectures.