Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Platt, Jason A.; Penny, Stephen G.; Smith, Timothy A.; Chen, Tse-Chun; Abarbanel, Henry I.

doi:10.1063/5.0156999

Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Journal Article · Tue Oct 03 00:00:00 EDT 2023 · Chaos: An Interdisciplinary Journal of Nonlinear Science

DOI:https://doi.org/10.1063/5.0156999· OSTI ID:2246611

^[1]; ^[2]; ^[3]; ^[4]; ^[5]

Univ. of California, San Diego, CA (United States)
Sofar Ocean, San Francisco, CA (United States); Univ. of Colorado, Boulder, CO (United States)
Univ. of Colorado, Boulder, CO (United States); National Oceanic and Atmospheric Administration (NOAA), Boulder, CO (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Univ. of California, San Diego, CA (United States). Scripps Inst. of Oceanography

Drawing on ergodic theory, we introduce a novel training method for machine learning based forecasting methods for chaotic dynamical systems. The training enforces dynamical invariants—such as the Lyapunov exponent spectrum and the fractal dimension—in the systems of interest, enabling longer and more stable forecasts when operating with limited data. The technique is demonstrated in detail using reservoir computing, a specific kind of recurrent neural network. Finally, results are given for the Lorenz 1996 chaotic dynamical system and a spectral quasi-geostrophic model of the atmosphere, both typical test cases for numerical weather prediction.

View Accepted Manuscript (DOE)

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 2246611

Report Number(s):: PNNL-SA--191042

Journal Information:: Chaos: An Interdisciplinary Journal of Nonlinear Science, Journal Name: Chaos: An Interdisciplinary Journal of Nonlinear Science Journal Issue: 10 Vol. 33; ISSN 1054-1500

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

References (39)

A Practical Guide to Applying Echo State Networks Lukoševičius, Mantas Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-35289-8_36	book	January 2012
The liapunov dimension of strange attractors Frederickson, Paul; Kaplan, James L.; Yorke, Ellen D. Journal of Differential Equations, Vol. 49, Issue 2 https://doi.org/10.1016/0022-0396(83)90011-6	journal	August 1983
A practical method for calculating largest Lyapunov exponents from small data sets Rosenstein, Michael T.; Collins, James J.; De Luca, Carlo J. Physica D: Nonlinear Phenomena, Vol. 65, Issue 1-2 https://doi.org/10.1016/0167-2789(93)90009-P	journal	May 1993
A robust method to estimate the maximal Lyapunov exponent of a time series Kantz, Holger Physics Letters A, Vol. 185, Issue 1 https://doi.org/10.1016/0375-9601(94)90991-1	journal	January 1994
Reservoir computing approaches to recurrent neural network training Lukoševičius, Mantas; Jaeger, Herbert Computer Science Review, Vol. 3, Issue 3 https://doi.org/10.1016/j.cosrev.2009.03.005	journal	August 2009
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data Yang, Liu; Meng, Xuhui; Karniadakis, George Em Journal of Computational Physics, Vol. 425 https://doi.org/10.1016/j.jcp.2020.109913	journal	January 2021
Physics-informed echo state networks Doan, N. A. K.; Polifke, W.; Magri, L. Journal of Computational Science, Vol. 47 https://doi.org/10.1016/j.jocs.2020.101237	journal	November 2020
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics Vlachas, P. R.; Pathak, J.; Hunt, B. R. Neural Networks, Vol. 126 https://doi.org/10.1016/j.neunet.2020.02.016	journal	June 2020
A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics Platt, Jason A.; Penny, Stephen G.; Smith, Timothy A. Neural Networks, Vol. 153 https://doi.org/10.1016/j.neunet.2022.06.025	journal	September 2022
Data‐Driven Equation Discovery of Ocean Mesoscale Closures Zanna, Laure; Bolton, Thomas Geophysical Research Letters, Vol. 47, Issue 17 https://doi.org/10.1029/2020GL088376	journal	August 2020
Integrating Recurrent Neural Networks With Data Assimilation for Scalable Data‐Driven State Estimation Penny, S. G.; Smith, T. A.; Chen, T. ‐C. Journal of Advances in Modeling Earth Systems, Vol. 14, Issue 3 https://doi.org/10.1029/2021MS002843	journal	March 2022
A cautionary tale for machine learning generated configurations in presence of a conserved quantity Azizi, Ahmadreza; Pleimling, Michel Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-85683-8	journal	March 2021
Physics-informed machine learning Karniadakis, George Em; Kevrekidis, Ioannis G.; Lu, Lu Nature Reviews Physics, Vol. 3, Issue 6 https://doi.org/10.1038/s42254-021-00314-5	journal	May 2021
Chaos in high-dimensional dissipative dynamical systems Ispolatov, Iaroslav; Madhok, Vaibhav; Allende, Sebastian Scientific Reports, Vol. 5, Issue 1 https://doi.org/10.1038/srep12506	journal	July 2015
Chaotic mixing in microdroplets Grigoriev, Roman O.; Schatz, Michael F.; Sharma, Vivek Lab on a Chip, Vol. 6, Issue 10 https://doi.org/10.1039/b607003e	journal	January 2006
Attractor reconstruction by machine learning Lu, Zhixin; Hunt, Brian R.; Ott, Edward Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28, Issue 6 https://doi.org/10.1063/1.5039508	journal	June 2018
Forecasting chaotic systems with very low connectivity reservoir computers Griffith, Aaron; Pomerance, Andrew; Gauthier, Daniel J. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 29, Issue 12 https://doi.org/10.1063/1.5120710	journal	December 2019
Robust forecasting using predictive generalized synchronization in reservoir computing Platt, Jason A.; Wong, Adrian; Clark, Randall Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, Issue 12 https://doi.org/10.1063/5.0066013	journal	December 2021
The general problem of the stability of motion Lyapunov, A. M. International Journal of Control, Vol. 55, Issue 3 https://doi.org/10.1080/00207179208934253	journal	March 1992
Short- and long-term predictions of chaotic flows and extreme events: a physics-constrained reservoir computing approach Doan, N. A. K.; Polifke, W.; Magri, L. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 477, Issue 2253 https://doi.org/10.1098/rspa.2021.0135	journal	September 2021
Introduction: mixing in microfluidics Ottino, Julio M.; Wiggins, Stephen Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, Vol. 362, Issue 1818 https://doi.org/10.1098/rsta.2003.1355	journal	May 2004
Kolmogorov entropy and numerical experiments Benettin, Giancarlo; Galgani, Luigi; Strelcyn, Jean-Marie Physical Review A, Vol. 14, Issue 6 https://doi.org/10.1103/PhysRevA.14.2338	journal	December 1976
Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems Beucler, Tom; Pritchard, Michael; Rasp, Stephan Physical Review Letters, Vol. 126, Issue 9 https://doi.org/10.1103/PhysRevLett.126.098302	journal	March 2021
Machine Learning Conservation Laws from Trajectories Liu, Ziming; Tegmark, Max Physical Review Letters, Vol. 126, Issue 18 https://doi.org/10.1103/PhysRevLett.126.180604	journal	May 2021
Experimental evidence of chaotic itinerancy and spatiotemporal chaos in optics Arecchi, F. T.; Giacomelli, G.; Ramazza, P. L. Physical Review Letters, Vol. 65, Issue 20 https://doi.org/10.1103/PhysRevLett.65.2531	journal	November 1990
Ergodic theory of chaos and strange attractors Eckmann, J. -P.; Ruelle, D. Reviews of Modern Physics, Vol. 57, Issue 3 https://doi.org/10.1103/RevModPhys.57.617	journal	July 1985
A New Approach to Linear Filtering and Prediction Problems Kalman, R. E. Journal of Basic Engineering, Vol. 82, Issue 1 https://doi.org/10.1115/1.3662552	journal	March 1960
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication Jaeger, H. Science, Vol. 304, Issue 5667 https://doi.org/10.1126/science.1091277	journal	April 2004
Learning skillful medium-range global weather forecasting Lam, Remi; Sanchez-Gonzalez, Alvaro; Willson, Matthew Science, Vol. 382, Issue 6677 https://doi.org/10.1126/science.adi2336	journal	December 2023
Comparison of Different Methods for Computing Lyapunov Exponents Geist, K.; Parlitz, U.; Lauterborn, W. Progress of Theoretical Physics, Vol. 83, Issue 5 https://doi.org/10.1143/PTP.83.875	journal	May 1990
Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations Maass, Wolfgang; Natschläger, Thomas; Markram, Henry Neural Computation, Vol. 14, Issue 11 https://doi.org/10.1162/089976602760407955	journal	November 2002
Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) Hansen, Nikolaus; Müller, Sibylle D.; Koumoutsakos, Petros Evolutionary Computation, Vol. 11, Issue 1 https://doi.org/10.1162/106365603321828970	journal	March 2003
Comparison of Local Ensemble Transform Kalman Filter, 3DVAR, and 4DVAR in a Quasigeostrophic Model Yang, Shu-Chih; Corazza, Matteo; Carrassi, Alberto Monthly Weather Review, Vol. 137, Issue 2 https://doi.org/10.1175/2008MWR2396.1	journal	February 2009
Finding Structure in Time Elman, Jeffrey L. Cognitive Science, Vol. 14, Issue 2 https://doi.org/10.1207/s15516709cog1402_1	journal	March 1990
Estimating fractal dimension Theiler, James Journal of the Optical Society of America A, Vol. 7, Issue 6 https://doi.org/10.1364/JOSAA.7.001055	journal	January 1990
qgs: A flexible Python framework of reduced-order multiscale climate models Demaeyer, Jonathan; De Cruz, Lesley; Vannitsem, Stéphane Journal of Open Source Software, Vol. 5, Issue 56 https://doi.org/10.21105/joss.02597	journal	December 2020
Four-dimensional variational assimilation and predictability in a quasi-geostrophic model Swanson, Kyle; Vautard, Robert; Pires, Carlos Tellus A: Dynamic Meteorology and Oceanography, Vol. 50, Issue 4 https://doi.org/10.3402/tellusa.v50i4.14540	journal	January 1998
Applications of Chaotic Dynamics in Robotics Zang, Xizhe; Iqbal, Sajid; Zhu, Yanhe International Journal of Advanced Robotic Systems, Vol. 13, Issue 2 https://doi.org/10.5772/62796	journal	March 2016

Similar Records

Chaos vs linear instability in the Vlasov equation: A fractal analysis characterization

Journal Article · Wed May 01 00:00:00 EDT 1996 · Physical Review, C · OSTI ID:283998

Divide and conquer: Learning chaotic dynamical systems with multistep penalty neural ordinary differential equations

Journal Article · Mon Oct 14 00:00:00 EDT 2024 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:3003857

Chaos, Ergodicity, and the Thermodynamics of Lower-Dimensional Time-Independent Hamiltonian Systems

Journal Article · Wed Aug 01 00:00:00 EDT 2001 · Phys.Rev.E · OSTI ID:1898838

Related Subjects

97 MATHEMATICS AND COMPUTING
Artificial intelligence
Artificial neural networks
Attractors
Chaotic systems
Ergodic theory
Lyapunov exponent
Machine learning
Theoretical and computational geophysics
Weather forecasting

Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Citation Formats

References (39)

Similar Records

Related Subjects