Neural ODE-RNNs: Continuous-Time Gated Dynamics

A modular PyTorch implementation and critical analysis of Neural Ordinary Differential Equation based Recurrent Neural Network Models. This project explores the fusion of gated RNN architectures (LSTM, GRU) with continuous-time Neural ODEs, aiming to replicate and evaluate the claims of rapid convergence and efficiency in handling irregularly sampled time series.

Learned vector field and trajectories for a spiral dynamical system.

🚀 Key Features

• Continuous-Time Gated Cells: Implementation of ODE-LSTM and ODE-GRU using continuous relaxation to bridge the gap between discrete gates and ODE dynamics.
• Curriculum Learning: A robust training strategy that gradually increases integration time, preventing gradient explosions and ensuring stable convergence on long sequences.
• Irregular Sampling Support: Natively handles non-uniform time intervals, making it ideal for real-world datasets like ICU monitoring (PhysioNet).
• Non-Linearity Stress Testing: Evaluation on the "Rounded Box" system ($y^3$ problem) to investigate the limits of MLP-based vector fields and the "exponentiation trick."
• Comprehensive Benchmarking: Tested against standard Neural ODEs and traditional RNNs on both synthetic (Lotka-Volterra, Fitzhugh-Nagumo) and real-world (HAR, PhysioNet) datasets.

---

🧠 Core Architectures

Continuous Relaxation

To solve the ambiguity of discrete gate notation in a continuous-time setting, we define a target state and move the current state towards it using learnable time constants $\tau$:

$$ \frac{dh}{dt} = \frac{h_{\text{target}}(h, x) - h}{\tau_h} $$ $$ \frac{dc}{dt} = \frac{c_{\text{target}}(h, x) - c}{\tau_c} $$

This ensures the system is always solving a well-defined Initial Value Problem (IVP).

Models Implemented

Model	Description
Neural ODE	Vanilla implementation with an MLP-parameterized vector field.
ODE-LSTM	Continuous relaxation of the LSTM cell with gated memory updates.
ODE-GRU	Continuous version of the GRU cell, optimized for parameter efficiency.

---

📊 Results & Insights

1. Curriculum Learning Impact

Integrating over full sequences early in training often leads to instability. Our ChunkScheduler allows the model to learn local dynamics first before tackling global trajectories.

MSE curves on the Spiral dataset with vs. without Curriculum Learning.

2. The $y^3$ Problem (Rounded Box)

Standard activation functions (Tanh/ReLU) struggle to approximate cubic non-linearities. We investigated the "exponentiation trick" and found it provides a significant architectural shortcut for specific systems.

3. Real-World Performance (HAR & PhysioNet)

Gated ODE variants show superior stability and final accuracy on complex tasks, particularly when data is sparse or irregularly sampled.

---

🛠️ Setup & Usage

Installation

1. Clone the repository:

   git clone https://github.com/LorenzoMioso/node-rnn.git
   cd node-rnn

2. Create a virtual environment:

   python3 -m venv venv
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

Running the Experiments

• Quick Demo: Train a Neural ODE on a spiral dataset:

  python train_neural_ode_spiral.py

• Reproducing Results: The core experiments are organized by paper section in the batch_scripts/ directory:

  • Section 2 (Regular Sampling): python batch_scripts/section_2_regular_sampling.py
  • Section 3 (Curriculum): python batch_scripts/section_3_curriculum_learning.py
  • Section 4 ($y^3$ Problem): python batch_scripts/section_4_y3_problem.py
  • Section 5 (HAR): python batch_scripts/section_5_har_classification.py
  • Section 6 (Irregular Sampling): python batch_scripts/section_6_irregular_sampling.py
  • Section 7 (PhysioNet): python batch_scripts/section_7_physionet.py

• Interactive Exploration: Explore the notebooks for step-by-step analysis:

  jupyter notebook main.ipynb

---

📂 Project Structure

• regular_sampling/: Implementations for fixed-interval datasets (HAR).
• irregular_sampling/: Implementations for non-uniform time series (PhysioNet).
• datasets/: Data generation and loading utilities for synthetic and real datasets.
• docs/: Analysis, paper (arXiv:2005.09807), and presentation materials.
• outputs/: Saved models, metrics, and generated figures.
• utils/: Helper functions for visualization, logging, and curriculum scheduling.

---

📜 References

This project is an implementation and extension of:

Neural Ordinary Differential Equation based Recurrent Neural Network Model
S. J. G. Lim et al. (2020)
arXiv:2005.09807

---

🤝 Acknowledgments

Developed by Lorenzo Mioso as a deep-dive into the intersections of continuous-time modeling and recurrent architectures.