Divide and conquer: Learning chaotic dynamical systems with multistep penalty neural ordinary differential equations

Chakraborty, Dibyajyoti; Chung, Seung Whan; Arcomano, Troy; Maulik, Romit

doi:10.1016/j.cma.2024.117442

Divide and conquer: Learning chaotic dynamical systems with multistep penalty neural ordinary differential equations

Journal Article · Mon Oct 14 00:00:00 EDT 2024 · Computer Methods in Applied Mechanics and Engineering

DOI:https://doi.org/10.1016/j.cma.2024.117442· OSTI ID:3003857

^[1]; ^[2]; Arcomano, Troy ^[3]; Maulik, Romit ^[4]

Pennsylvania State Univ., University Park, PA (United States)
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States)
Pennsylvania State Univ., University Park, PA (United States); Argonne National Laboratory (ANL), Argonne, IL (United States)

Forecasting high-dimensional dynamical systems is a fundamental challenge in various fields, such as geosciences and engineering. Neural Ordinary Differential Equations (NODEs), which combine the power of neural networks and numerical solvers, have emerged as a promising algorithm for forecasting complex nonlinear dynamical systems. However, classical techniques used for NODE training are ineffective for learning chaotic dynamical systems. In this work, we propose a novel NODE-training approach that allows for robust learning of chaotic dynamical systems. Here, our method addresses the challenges of non-convexity and exploding gradients associated with underlying chaotic dynamics. Training data trajectories from such systems are split into multiple, non-overlapping time windows. In addition to the deviation from the training data, the optimization loss term further penalizes the discontinuities of the predicted trajectory between the time windows. The window size is selected based on the fastest Lyapunov time scale of the system. Multi-step penalty(MP) method is first demonstrated on Lorenz equation, to illustrate how it improves the loss landscape and thereby accelerates the optimization convergence. MP method can optimize chaotic systems in a manner similar to least-squares shadowing with significantly lower computational costs. Our proposed algorithm, denoted the Multistep Penalty NODE, is applied to chaotic systems such as the Kuramoto-Sivashinsky equation, the two-dimensional Kolmogorov flow, and ERA5 reanalysis data for the atmosphere. It is observed that MP-NODE provide viable performance for such chaotic systems, not only for short-term trajectory predictions but also for invariant statistics that are hallmarks of the chaotic nature of these dynamics.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); Argonne National Laboratory - Laboratory Directed Research and Development (LDRD); USDOE Office of Science (SC), Fusion Energy Sciences (FES)

Grant/Contract Number:: AC52-07NA27344; AC02-06CH11357

OSTI ID:: 3003857

Journal Information:: Computer Methods in Applied Mechanics and Engineering, Journal Name: Computer Methods in Applied Mechanics and Engineering Vol. 432; ISSN 0045-7825

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (38)

The ERA5 global reanalysis Hersbach, Hans; Bell, Bill; Berrisford, Paul Quarterly Journal of the Royal Meteorological Society, Vol. 146, Issue 730 https://doi.org/10.1002/qj.3803	journal	June 2020
Order and complexity in the Kuramoto-Sivashinsky model of weakly turbulent interfaces Hyman, James M.; Nicolaenko, Basil; Zaleski, Stéphane Physica D: Nonlinear Phenomena, Vol. 23, Issue 1-3 https://doi.org/10.1016/0167-2789(86)90136-3	journal	December 1986
A Multiple Shooting Algorithm for Direct Solution of Optimal Control Problems * Bock, H. G.; Plitt, K. J. IFAC Proceedings Volumes, Vol. 17, Issue 2 https://doi.org/10.1016/S1474-6670(17)61205-9	journal	July 1984
FDOT: A Fast, memory-efficient and automated approach for Discrete adjoint sensitivity analysis using the Operator overloading Technique Djeddi, Reza; Ekici, Kivanc Aerospace Science and Technology, Vol. 91 https://doi.org/10.1016/j.ast.2019.05.004	journal	August 2019
Runge–Kutta pairs of order 5(4) satisfying only the first column simplifying assumption Tsitouras, Ch. Computers & Mathematics with Applications, Vol. 62, Issue 2 https://doi.org/10.1016/j.camwa.2011.06.002	journal	July 2011
Adjoint sensitivity analysis of chaotic systems using cumulant truncation Craske, John Chaos, Solitons & Fractals, Vol. 119 https://doi.org/10.1016/j.chaos.2018.12.024	journal	February 2019
Least Squares Shadowing sensitivity analysis of chaotic limit cycle oscillations Wang, Qiqi; Hu, Rui; Blonigan, Patrick Journal of Computational Physics, Vol. 267 https://doi.org/10.1016/j.jcp.2014.03.002	journal	June 2014
Sensitivity analysis on chaotic dynamical systems by Non-Intrusive Least Squares Shadowing (NILSS) Ni, Angxiu; Wang, Qiqi Journal of Computational Physics, Vol. 347 https://doi.org/10.1016/j.jcp.2017.06.033	journal	October 2017
Adjoint sensitivity analysis on chaotic dynamical systems by Non-Intrusive Least Squares Adjoint Shadowing (NILSAS) Ni, Angxiu; Talnikar, Chaitanya Journal of Computational Physics, Vol. 395 https://doi.org/10.1016/j.jcp.2019.06.035	journal	October 2019
An optimization method for chaotic turbulent flow Chung, Seung Whan; Freund, Jonathan B. Journal of Computational Physics, Vol. 457 https://doi.org/10.1016/j.jcp.2022.111077	journal	May 2022
Stabilized neural ordinary differential equations for long-time forecasting of dynamical systems Linot, Alec J.; Burby, Joshua W.; Tang, Qi Journal of Computational Physics, Vol. 474 https://doi.org/10.1016/j.jcp.2022.111838	journal	February 2023
Time-series learning of latent-space dynamics for reduced-order model closure Maulik, Romit; Mohan, Arvind; Lusch, Bethany Physica D: Nonlinear Phenomena, Vol. 405 https://doi.org/10.1016/j.physd.2020.132368	journal	April 2020
Assessments of epistemic uncertainty using Gaussian stochastic weight averaging for fluid-flow regression Morimoto, Masaki; Fukami, Kai; Maulik, Romit Physica D: Nonlinear Phenomena, Vol. 440 https://doi.org/10.1016/j.physd.2022.133454	journal	November 2022
Accurate medium-range global weather forecasting with 3D neural networks Bi, Kaifeng; Xie, Lingxi; Zhang, Hengheng Nature, Vol. 619, Issue 7970 https://doi.org/10.1038/s41586-023-06185-3	journal	July 2023
Stiff neural ordinary differential equations Kim, Suyong; Ji, Weiqi; Deng, Sili Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, Issue 9 https://doi.org/10.1063/5.0060697	journal	September 2021
Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations Linot, Alec J.; Graham, Michael D. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 32, Issue 7 https://doi.org/10.1063/5.0069536	journal	July 2022
Stability preserving data-driven models with latent dynamics Luo, Yushuang; Li, Xiantao; Hao, Wenrui Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 32, Issue 8 https://doi.org/10.1063/5.0096889	journal	August 2022
Machine learning–accelerated computational fluid dynamics Kochkov, Dmitrii; Smith, Jamie A.; Alieva, Ayya Proceedings of the National Academy of Sciences, Vol. 118, Issue 21 https://doi.org/10.1073/pnas.2101784118	journal	May 2021
DISCRETE- vs. CONTINUOUS-TIME NONLINEAR SIGNAL PROCESSING OF Cu ELECTRODISSOLUTION DATA Rico-MartÍNez, R.; Krischer, K.; Kevrekidis, I. G. Chemical Engineering Communications, Vol. 118, Issue 1 https://doi.org/10.1080/00986449208936084	journal	November 1992
Ruelle's linear response formula, ensemble adjoint schemes and Lévy flights Eyink, G. L.; Haine, T. W. N.; Lea, D. J. Nonlinearity, Vol. 17, Issue 5 https://doi.org/10.1088/0951-7715/17/5/016	journal	July 2004
Neural partial differential equations for chaotic systems Gelbrecht, Maximilian; Boers, Niklas; Kurths, Jürgen New Journal of Physics, Vol. 23, Issue 4 https://doi.org/10.1088/1367-2630/abeb90	journal	April 2021
Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach Pathak, Jaideep; Hunt, Brian; Girvan, Michelle Physical Review Letters, Vol. 120, Issue 2 https://doi.org/10.1103/PhysRevLett.120.024102	journal	January 2018
Learning long-term dependencies with gradient descent is difficult Bengio, Y.; Simard, P.; Frasconi, P. IEEE Transactions on Neural Networks, Vol. 5, Issue 2 https://doi.org/10.1109/72.279181	journal	March 1994
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Multiple Shooting for Training Neural Differential Equations on Time Series Turan, Evren Mert; Jaschke, Johannes IEEE Control Systems Letters, Vol. 6 https://doi.org/10.1109/LCSYS.2021.3135835	journal	January 2022
Solving Ordinary Differential Equations I: Nonstiff Problems (E. Hairer, S. P. Norsett, and G. Wanner) Alexander, Roger SIAM Review, Vol. 32, Issue 3 https://doi.org/10.1137/1032091	journal	September 1990
Fast Adjoint Algorithm for Linear Responses of Hyperbolic Chaos Ni, Angxiu SIAM Journal on Applied Dynamical Systems, Vol. 22, Issue 4 https://doi.org/10.1137/22M1522383	journal	October 2023
Nonlinear Dynamical System Identification from Uncertain and Indirect Measurements Voss, Henning U.; Timmer, Jens; Kurths, JÜRgen International Journal of Bifurcation and Chaos, Vol. 14, Issue 06 https://doi.org/10.1142/S0218127404010345	journal	June 2004
Graph Sequential Neural ODE Process for Link Prediction on Dynamic and Sparse Graphs Luo, Linhao; Haffari, Gholamreza; Pan, Shirui Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining https://doi.org/10.1145/3539597.3570465	conference	February 2023
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks Williams, Ronald J.; Zipser, David Neural Computation, Vol. 1, Issue 2 https://doi.org/10.1162/neco.1989.1.2.270	journal	June 1989
Long Short-Term Memory Hochreiter, Sepp; Schmidhuber, Jürgen Neural Computation, Vol. 9, Issue 8 https://doi.org/10.1162/neco.1997.9.8.1735	journal	November 1997
Gated Orthogonal Recurrent Units: On Learning to Forget Jing, Li; Gulcehre, Caglar; Peurifoy, John Neural Computation, Vol. 31, Issue 4 https://doi.org/10.1162/neco_a_01174	journal	April 2019
Sensitivity analysis of the climate of a chaotic ocean circulation model Lea, Daniel J.; Haine, Thomas W. N.; Allen, Myles R. Quarterly Journal of the Royal Meteorological Society, Vol. 128, Issue 586 https://doi.org/10.1256/qj.01.180	journal	October 2002
Climate sensitivities via a Fokker–Planck adjoint approach Thuburn, J. Quarterly Journal of the Royal Meteorological Society, Vol. 131, Issue 605 https://doi.org/10.1256/qj.04.46	journal	January 2005
Least-Squares Shadowing Sensitivity Analysis of Chaotic Flow Around a Two-Dimensional Airfoil Blonigan, Patrick J.; Wang, Qiqi; Nielsen, Eric J. AIAA Journal, Vol. 56, Issue 2 https://doi.org/10.2514/1.J055389	journal	February 2018
Least Squares Shadowing for Sensitivity Analysis of Turbulent Fluid Flows Blonigan, Patrick J.; Gomez, Steven A.; Wang, Qiqi 52nd Aerospace Sciences Meeting https://doi.org/10.2514/6.2014-1426	conference	January 2014
An Analysis of the Ensemble Adjoint Approach to Sensitivity Analysis in Chaotic Systems Chandramoorthy, Nisha; Fernandez, Pablo; Talnikar, Chaitanya 23rd AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2017-3799	conference	June 2017
Improving the Adversarial Robustness of Neural ODE Image Classifiers by Tuning the Tolerance Parameter Carrara, Fabio; Caldelli, Roberto; Falchi, Fabrizio Information, Vol. 13, Issue 12 https://doi.org/10.3390/info13120555	journal	November 2022

Similar Records

Understanding latent timescales in neural ordinary differential equation models of advection-dominated dynamical systems

Journal Article · Tue Apr 01 00:00:00 EDT 2025 · Physica. D, Nonlinear Phenomena · OSTI ID:3004189

Learning subgrid-scale models with neural ordinary differential equations

Journal Article · Sat May 06 00:00:00 EDT 2023 · Computers and Fluids · OSTI ID:2377337

Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Journal Article · Tue Oct 03 00:00:00 EDT 2023 · Chaos: An Interdisciplinary Journal of Nonlinear Science · OSTI ID:2246611

Related Subjects

Chaotic dynamical systems
Neural ordinary differential equations
Scientific machine learning
machine learning

Divide and conquer: Learning chaotic dynamical systems with multistep penalty neural ordinary differential equations

Citation Formats

References (38)

Similar Records

Related Subjects