Stiff neural ordinary differential equations

Kim, Suyong; Ji, Weiqi; Deng, Sili; Ma, Yingbo; Rackauckas, Christopher

doi:10.1063/5.0060697

Stiff neural ordinary differential equations

Journal Article · Mon Sep 20 00:00:00 EDT 2021 · Chaos: An Interdisciplinary Journal of Nonlinear Science

DOI:https://doi.org/10.1063/5.0060697· OSTI ID:1848383

^[1]; ^[2]; ^[2]; Ma, Yingbo ^[3]; Rackauckas, Christopher ^[4]

Massachusetts Institute of Technology, Cambridge, MA (United States). Department of Mechanical Engineering; Massachusetts Inst. of Tech., Cambridge, MA (United States)
Massachusetts Institute of Technology, Cambridge, MA (United States). Department of Mechanical Engineering
Julia Computing Inc., Cambridge, MA (United States)
Massachusetts Institute of Technology, Cambridge, MA (United States). Department of Mechanical Engineering; University of Maryland, Baltimore, MD (United States). School of Pharmacy; Pumas AI, Baltimore, MD (United States)

Neural Ordinary Differential Equations (ODEs) are a promising approach to learn dynamical models from time-series data in science and engineering applications. This work aims at learning neural ODEs for stiff systems, which are usually raised from chemical kinetic modeling in chemical and biological systems. We first show the challenges of learning neural ODEs in the classical stiff ODE systems of Robertson’s problem and propose techniques to mitigate the challenges associated with scale separations in stiff systems. We then present successful demonstrations in stiff systems of Robertson’s problem and an air pollution problem. The demonstrations show that the usage of deep networks with rectified activations, proper scaling of the network outputs as well as loss functions, and stabilized gradient calculations are the key techniques enabling the learning of stiff neural ODEs. The success of learning stiff neural ODEs opens up possibilities of using neural ODEs in applications with widely varying time-scales, such as chemical dynamics in energy conversion, environmental engineering, and life sciences.

View Accepted Manuscript (DOE)

Research Organization:: Julia Computing, Cambridge, MA (United States)

Sponsoring Organization:: USDOE Advanced Research Projects Agency - Energy (ARPA-E)

Grant/Contract Number:: AR0001222

OSTI ID:: 1848383

Journal Information:: Chaos: An Interdisciplinary Journal of Nonlinear Science, Journal Name: Chaos: An Interdisciplinary Journal of Nonlinear Science Journal Issue: 9 Vol. 31; ISSN 1054-1500

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

References (24)

Convergence analysis of Krylov subspace methods: Convergence analysis of Krylov subspace methods Liesen, Jörg; Tichý, Petr GAMM-Mitteilungen, Vol. 27, Issue 2 https://doi.org/10.1002/gamm.201490008	journal	December 2004
DiffEqFlux.jl - A Julia Library for Neural Differential Equations Rackauckas, Chris; Innes, Mike; Ma, Yingbo arXiv https://doi.org/10.48550/arxiv.1902.02376	preprint	January 2019
Time-series learning of latent-space dynamics for reduced-order model closure Maulik, Romit; Mohan, Arvind; Lusch, Bethany arXiv https://doi.org/10.48550/arxiv.1906.07815	text	January 2019
Latent ODEs for Irregularly-Sampled Time Series Rubanova, Yulia; Chen, Ricky T. Q.; Duvenaud, David arXiv https://doi.org/10.48550/arxiv.1907.03907	preprint	January 2019
Universal Differential Equations for Scientific Machine Learning Rackauckas, Christopher; Ma, Yingbo; Martensen, Julius arXiv https://doi.org/10.48550/arxiv.2001.04385	preprint	January 2020
Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks Anantharaman, Ranjan; Ma, Yingbo; Gowda, Shashi arXiv https://doi.org/10.48550/arxiv.2010.04004	preprint	January 2020
CellBox: Interpretable Machine Learning for Perturbation Biology with Application to the Design of Cancer Combination Therapy Yuan, Bo; Shen, Ciyue; Luna, Augustin Cell Systems, Vol. 12, Issue 2 https://doi.org/10.1016/j.cels.2020.11.013	journal	February 2021
Time-series learning of latent-space dynamics for reduced-order model closure Maulik, Romit; Mohan, Arvind; Lusch, Bethany Physica D: Nonlinear Phenomena, Vol. 405 https://doi.org/10.1016/j.physd.2020.132368	journal	April 2020
Autonomous Discovery of Unknown Reaction Pathways from Data by Chemical Reaction Neural Network Ji, Weiqi; Deng, Sili The Journal of Physical Chemistry A, Vol. 125, Issue 4 https://doi.org/10.1021/acs.jpca.0c09316	journal	January 2021
Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics Ji, Weiqi; Qiu, Weilun; Shi, Zhiyu The Journal of Physical Chemistry A, Vol. 125, Issue 36 https://doi.org/10.1021/acs.jpca.1c05102	journal	August 2021
Reactive SINDy: Discovering governing reactions from concentration data Hoffmann, Moritz; Fröhner, Christoph; Noé, Frank The Journal of Chemical Physics, Vol. 150, Issue 2 https://doi.org/10.1063/1.5066099	journal	January 2019
Discovering governing equations from data by sparse identification of nonlinear dynamical systems Brunton, Steven L.; Proctor, Joshua L.; Kutz, J. Nathan Proceedings of the National Academy of Sciences, Vol. 113, Issue 15 https://doi.org/10.1073/pnas.1517384113	journal	March 2016
Inferring Biological Networks by Sparse Identification of Nonlinear Dynamics Mangan, Niall M.; Brunton, Steven L.; Proctor, Joshua L. IEEE Transactions on Molecular, Biological and Multi-Scale Communications, Vol. 2, Issue 1 https://doi.org/10.1109/TMBMC.2016.2633265	journal	June 2016
Differential/Algebraic Equations are not ODE’ s Petzold, Linda SIAM Journal on Scientific and Statistical Computing, Vol. 3, Issue 3 https://doi.org/10.1137/0903023	journal	September 1982
Gauss–Seidel Iteration for Stiff ODES from Chemical Kinetics Verwer, J. G. SIAM Journal on Scientific Computing, Vol. 15, Issue 5 https://doi.org/10.1137/0915076	journal	September 1994
A User’s View of Solving Stiff Ordinary Differential Equations Shampine, L. F.; Gear, C. W. SIAM Review, Vol. 21, Issue 1 https://doi.org/10.1137/1021001	journal	January 1979
FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of ODEs Zhang, Hong; Sandu, Adrian SIAM Journal on Scientific Computing, Vol. 36, Issue 5 https://doi.org/10.1137/130912335	journal	January 2014
Julia: A Fresh Approach to Numerical Computing Bezanson, Jeff; Edelman, Alan; Karpinski, Stefan SIAM Review, Vol. 59, Issue 1 https://doi.org/10.1137/141000671	journal	January 2017
Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks Wang, Sifan; Teng, Yujun; Perdikaris, Paris SIAM Journal on Scientific Computing, Vol. 43, Issue 5 https://doi.org/10.1137/20M1318043	journal	January 2021
SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers Hindmarsh, Alan C.; Brown, Peter N.; Grant, Keith E. ACM Transactions on Mathematical Software, Vol. 31, Issue 3 https://doi.org/10.1145/1089014.1089020	journal	September 2005
Flux: Elegant machine learning with Julia Innes, Mike Journal of Open Source Software, Vol. 3, Issue 25 https://doi.org/10.21105/joss.00602	journal	May 2018
Universal Differential Equations for Scientific Machine Learning Rackauckas, Christopher; Ma, Yingbo; Martensen, Julius https://doi.org/10.21203/rs.3.rs-55125/v1	preprint	August 2020
Stiff systems Shampine, Lawrence; Thompson, Skip Scholarpedia, Vol. 2, Issue 3 https://doi.org/10.4249/scholarpedia.2855	journal	January 2007
DifferentialEquations.jl – A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia Rackauckas, Christopher; Nie, Qing Journal of Open Research Software, Vol. 5 https://doi.org/10.5334/jors.151	journal	May 2017

Similar Records

Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems

Technical Report · Thu Nov 05 23:00:00 EST 2020 · OSTI ID:1706214

Learning subgrid-scale models with neural ordinary differential equations

Journal Article · Sat May 06 00:00:00 EDT 2023 · Computers and Fluids · OSTI ID:2377337

Dynamic Parameter Estimation with Physics-based Neural Ordinary Differential Equations

Conference · Sun Jul 17 00:00:00 EDT 2022 · 2022 IEEE Power & Energy Society General Meeting (PESGM) · OSTI ID:1958412

Related Subjects

97 MATHEMATICS AND COMPUTING
Air pollution
Artificial neural networks
Chemical dynamics
Data science
Dynamical systems
Energy conversion
Machine learning
Mathematics
Numerical differentiation
Optimization algorithms
Physics

Stiff neural ordinary differential equations

Citation Formats

References (24)

Similar Records

Related Subjects