Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Platt, Jason A.; Penny, Stephen G.; Smith, Timothy A.; Chen, Tse-Chun; Abarbanel, Henry I.

doi:10.1063/5.0156999

Title: Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Journal Article · Tue Oct 03 00:00:00 EDT 2023 · Chaos: An Interdisciplinary Journal of Nonlinear Science

DOI:https://doi.org/10.1063/5.0156999· OSTI ID:2246611

^[1];

^[2];

^[3];

^[4];

^[5]

Univ. of California, San Diego, CA (United States)
Sofar Ocean, San Francisco, CA (United States); Univ. of Colorado, Boulder, CO (United States)
Univ. of Colorado, Boulder, CO (United States); National Oceanic and Atmospheric Administration (NOAA), Boulder, CO (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Univ. of California, San Diego, CA (United States). Scripps Inst. of Oceanography

Drawing on ergodic theory, we introduce a novel training method for machine learning based forecasting methods for chaotic dynamical systems. The training enforces dynamical invariants—such as the Lyapunov exponent spectrum and the fractal dimension—in the systems of interest, enabling longer and more stable forecasts when operating with limited data. The technique is demonstrated in detail using reservoir computing, a specific kind of recurrent neural network. Finally, results are given for the Lorenz 1996 chaotic dynamical system and a spectral quasi-geostrophic model of the atmosphere, both typical test cases for numerical weather prediction.

This content will become publicly available on Thu Oct 03 00:00:00 EDT 2024

Cite

Export

Save

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 2246611

Report Number(s):: PNNL-SA-191042

Journal Information:: Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 33, Issue 10; ISSN 1054-1500

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

References (39)

Comparison of Local Ensemble Transform Kalman Filter, 3DVAR, and 4DVAR in a Quasigeostrophic Model Yang, Shu-Chih; Corazza, Matteo; Carrassi, Alberto Monthly Weather Review, Vol. 137, Issue 2 https://doi.org/10.1175/2008MWR2396.1	journal	February 2009
Physics-informed echo state networks Doan, N. A. K.; Polifke, W.; Magri, L. Journal of Computational Science, Vol. 47 https://doi.org/10.1016/j.jocs.2020.101237	journal	November 2020
qgs: A flexible Python framework of reduced-order multiscale climate models Demaeyer, Jonathan; De Cruz, Lesley; Vannitsem, Stéphane Journal of Open Source Software, Vol. 5, Issue 56 https://doi.org/10.21105/joss.02597	journal	December 2020
Short- and long-term predictions of chaotic flows and extreme events: a physics-constrained reservoir computing approach Doan, N. A. K.; Polifke, W.; Magri, L. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 477, Issue 2253 https://doi.org/10.1098/rspa.2021.0135	journal	September 2021
Integrating Recurrent Neural Networks With Data Assimilation for Scalable Data‐Driven State Estimation Penny, S. G.; Smith, T. A.; Chen, T. ‐C. Journal of Advances in Modeling Earth Systems, Vol. 14, Issue 3 https://doi.org/10.1029/2021MS002843	journal	March 2022
Physics-informed machine learning Karniadakis, George Em; Kevrekidis, Ioannis G.; Lu, Lu Nature Reviews Physics, Vol. 3, Issue 6 https://doi.org/10.1038/s42254-021-00314-5	journal	May 2021
A cautionary tale for machine learning generated configurations in presence of a conserved quantity Azizi, Ahmadreza; Pleimling, Michel Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-85683-8	journal	March 2021
Forecasting chaotic systems with very low connectivity reservoir computers Griffith, Aaron; Pomerance, Andrew; Gauthier, Daniel J. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 29, Issue 12 https://doi.org/10.1063/1.5120710	journal	December 2019
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
Attractor reconstruction by machine learning Lu, Zhixin; Hunt, Brian R.; Ott, Edward Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28, Issue 6 https://doi.org/10.1063/1.5039508	journal	June 2018
Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication Jaeger, H. Science, Vol. 304, Issue 5667 https://doi.org/10.1126/science.1091277	journal	April 2004
A robust method to estimate the maximal Lyapunov exponent of a time series Kantz, Holger Physics Letters A, Vol. 185, Issue 1 https://doi.org/10.1016/0375-9601(94)90991-1	journal	January 1994
Machine Learning Conservation Laws from Trajectories Liu, Ziming; Tegmark, Max Physical Review Letters, Vol. 126, Issue 18 https://doi.org/10.1103/PhysRevLett.126.180604	journal	May 2021
Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems Beucler, Tom; Pritchard, Michael; Rasp, Stephan Physical Review Letters, Vol. 126, Issue 9 https://doi.org/10.1103/PhysRevLett.126.098302	journal	March 2021
Chaotic mixing in microdroplets Grigoriev, Roman O.; Schatz, Michael F.; Sharma, Vivek Lab on a Chip, Vol. 6, Issue 10 https://doi.org/10.1039/b607003e	journal	January 2006
Applications of Chaotic Dynamics in Robotics Zang, Xizhe; Iqbal, Sajid; Zhu, Yanhe International Journal of Advanced Robotic Systems, Vol. 13, Issue 2 https://doi.org/10.5772/62796	journal	March 2016
Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) Hansen, Nikolaus; Müller, Sibylle D.; Koumoutsakos, Petros Evolutionary Computation, Vol. 11, Issue 1 https://doi.org/10.1162/106365603321828970	journal	March 2003
Introduction: mixing in microfluidics Ottino, Julio M.; Wiggins, Stephen Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, Vol. 362, Issue 1818 https://doi.org/10.1098/rsta.2003.1355	journal	May 2004
Reservoir computing approaches to recurrent neural network training Lukoševičius, Mantas; Jaeger, Herbert Computer Science Review, Vol. 3, Issue 3 https://doi.org/10.1016/j.cosrev.2009.03.005	journal	August 2009
Finding Structure in Time Elman, Jeffrey L. Cognitive Science, Vol. 14, Issue 2 https://doi.org/10.1207/s15516709cog1402_1	journal	March 1990
B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data Yang, Liu; Meng, Xuhui; Karniadakis, George Em Journal of Computational Physics, Vol. 425 https://doi.org/10.1016/j.jcp.2020.109913	journal	January 2021
Comparison of Different Methods for Computing Lyapunov Exponents Geist, K.; Parlitz, U.; Lauterborn, W. Progress of Theoretical Physics, Vol. 83, Issue 5 https://doi.org/10.1143/PTP.83.875	journal	May 1990
A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics Platt, Jason A.; Penny, Stephen G.; Smith, Timothy A. Neural Networks, Vol. 153 https://doi.org/10.1016/j.neunet.2022.06.025	journal	September 2022
Data‐Driven Equation Discovery of Ocean Mesoscale Closures Zanna, Laure; Bolton, Thomas Geophysical Research Letters, Vol. 47, Issue 17 https://doi.org/10.1029/2020GL088376	journal	August 2020
The general problem of the stability of motion Lyapunov, A. M. International Journal of Control, Vol. 55, Issue 3 https://doi.org/10.1080/00207179208934253	journal	March 1992
Robust forecasting using predictive generalized synchronization in reservoir computing Platt, Jason A.; Wong, Adrian; Clark, Randall Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, Issue 12 https://doi.org/10.1063/5.0066013	journal	December 2021
A Practical Guide to Applying Echo State Networks Lukoševičius, Mantas Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-35289-8_36	book	January 2012
A New Approach to Linear Filtering and Prediction Problems Kalman, R. E. Journal of Basic Engineering, Vol. 82, Issue 1 https://doi.org/10.1115/1.3662552	journal	March 1960
Estimating fractal dimension Theiler, James Journal of the Optical Society of America A, Vol. 7, Issue 6 https://doi.org/10.1364/JOSAA.7.001055	journal	January 1990
A practical method for calculating largest Lyapunov exponents from small data sets Rosenstein, Michael T.; Collins, James J.; De Luca, Carlo J. Physica D: Nonlinear Phenomena, Vol. 65, Issue 1-2 https://doi.org/10.1016/0167-2789(93)90009-P	journal	May 1993
Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations Maass, Wolfgang; Natschläger, Thomas; Markram, Henry Neural Computation, Vol. 14, Issue 11 https://doi.org/10.1162/089976602760407955	journal	November 2002
Chaos in high-dimensional dissipative dynamical systems Ispolatov, Iaroslav; Madhok, Vaibhav; Allende, Sebastian Scientific Reports, Vol. 5, Issue 1 https://doi.org/10.1038/srep12506	journal	July 2015
The liapunov dimension of strange attractors Frederickson, Paul; Kaplan, James L.; Yorke, Ellen D. Journal of Differential Equations, Vol. 49, Issue 2 https://doi.org/10.1016/0022-0396(83)90011-6	journal	August 1983
Experimental evidence of chaotic itinerancy and spatiotemporal chaos in optics Arecchi, F. T.; Giacomelli, G.; Ramazza, P. L. Physical Review Letters, Vol. 65, Issue 20 https://doi.org/10.1103/PhysRevLett.65.2531	journal	November 1990
Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics Vlachas, P. R.; Pathak, J.; Hunt, B. R. Neural Networks, Vol. 126 https://doi.org/10.1016/j.neunet.2020.02.016	journal	June 2020
Kolmogorov entropy and numerical experiments Benettin, Giancarlo; Galgani, Luigi; Strelcyn, Jean-Marie Physical Review A, Vol. 14, Issue 6 https://doi.org/10.1103/PhysRevA.14.2338	journal	December 1976
Ergodic theory of chaos and strange attractors Eckmann, J. -P.; Ruelle, D. Reviews of Modern Physics, Vol. 57, Issue 3 https://doi.org/10.1103/RevModPhys.57.617	journal	July 1985
Four-dimensional variational assimilation and predictability in a quasi-geostrophic model Swanson, Kyle; Vautard, Robert; Pires, Carlos Tellus A: Dynamic Meteorology and Oceanography, Vol. 50, Issue 4 https://doi.org/10.3402/tellusa.v50i4.14540	journal	January 1998
Learning skillful medium-range global weather forecasting Lam, Remi; Sanchez-Gonzalez, Alvaro; Willson, Matthew Science, Vol. 382, Issue 6677 https://doi.org/10.1126/science.adi2336	journal	December 2023

Similar Records

Resonances in a Chaotic Attractor Crisis of the Lorenz Flow

Journal Article · Thu Feb 15 00:00:00 EST 2018 · Journal of Statistical Physics · OSTI ID:2246611

Tantet, Alexis; Lucarini, Valerio; Dijkstra, Henk A.

Neural Network Based Montioring and Control of Fluidized Bed.

Technical Report · Mon Apr 01 00:00:00 EST 1996 · OSTI ID:2246611

Bodruzzaman, M; Essawy, M A

A Non‐Intrusive Machine Learning Framework for Debiasing Long‐Time Coarse Resolution Climate Simulations and Quantifying Rare Events Statistics

Journal Article · Tue Mar 26 00:00:00 EDT 2024 · Journal of Advances in Modeling Earth Systems · OSTI ID:2246611

Barthel Sorensen, B.; Charalampopoulos, A.; Zhang, S.; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Artificial intelligence
Machine learning
Lyapunov exponent
Attractors
Chaotic systems
Theoretical and computational geophysics
Weather forecasting
Artificial neural networks
Ergodic theory

Title: Constraining chaos: Enforcing dynamical invariants in the training of reservoir computers

Citation Formats

References (39)

Similar Records

Related Subjects