Regression-Based Projection for Learning Mori–Zwanzig Operators

Lin, Yen Ting; Tian, Yifeng; Perez, Danny; Livescu, Daniel

doi:10.1137/22m1506146

Regression-Based Projection for Learning Mori–Zwanzig Operators

Journal Article · Fri Oct 13 00:00:00 EDT 2023 · SIAM Journal on Applied Dynamical Systems

DOI:https://doi.org/10.1137/22m1506146· OSTI ID:2203394

^[1]; ^[1]; ^[1]; ^[1]

Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

We propose to adopt statistical regression as the projection operator to enable data-driven learning of the operators in the Mori–Zwanzig formalism. We present a principled method to extract the Markov and memory operators for any regression models. We show that the choice of linear regression results in a recently proposed data-driven learning algorithm based on Mori’s projection operator, which is a higher-order approximate Koopman learning method. We show that more expressive nonlinear regression models naturally fill in the gap between the highly idealized and computationally efficient Mori’s projection operator and the most optimal yet computationally infeasible Zwanzig’s projection operator. We performed numerical experiments and extracted the operators for an array of regression-based projections, including linear, polynomial, spline, and neural network–based regressions, showing a progressive improvement as the complexity of the regression model increased. In conclusion, our proposition provides a general framework to extract memory-dependent corrections and can be readily applied to an array of data-driven learning methods for stationary dynamical systems in the literature.

View Accepted Manuscript (DOE)

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001

OSTI ID:: 2203394

Alternate ID(s):: OSTI ID: 2202613
OSTI ID: 2202615

Report Number(s):: LA-UR--22-24323

Journal Information:: SIAM Journal on Applied Dynamical Systems, Journal Name: SIAM Journal on Applied Dynamical Systems Journal Issue: 4 Vol. 22; ISSN 1536-0040

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (50)

A Numerical Procedure to Evaluate Memory Effects in Non‐Equilibrium Coarse‐Grained Models Meyer, Hugues; Wolf, Steffen; Stock, Gerhard Advanced Theory and Simulations, Vol. 4, Issue 4 https://doi.org/10.1002/adts.202000197	journal	November 2020
Regression and Projection Afriat, S. N. Economic Models, Estimation and Risk Programming: Essays in Honor of Gerhard Tintner https://doi.org/10.1007/978-3-642-46198-9_12	book	January 1969
Nonlinear generalized Langevin equations Zwanzig, Robert Journal of Statistical Physics, Vol. 9, Issue 3 https://doi.org/10.1007/BF01008729	journal	November 1973
Detecting strange attractors in turbulence Takens, Floris Lecture Notes in Mathematics https://doi.org/10.1007/BFb0091924	book	January 1981
Applications of the dynamic mode decomposition Schmid, P. J.; Li, L.; Juniper, M. P. Theoretical and Computational Fluid Dynamics, Vol. 25, Issue 1-4, p. 249-259 https://doi.org/10.1007/s00162-010-0203-9	journal	August 2010
A Data–Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition Williams, Matthew O.; Kevrekidis, Ioannis G.; Rowley, Clarence W. Journal of Nonlinear Science, Vol. 25, Issue 6 https://doi.org/10.1007/s00332-015-9258-5	journal	June 2015
Die verallgemeinerte Liouville-Gleichung Gerlich, G. Physica, Vol. 69, Issue 2 https://doi.org/10.1016/0031-8914(73)90083-9	journal	November 1973
Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations Sivashinsky, G. I. Acta Astronautica, Vol. 4, Issue 11-12 https://doi.org/10.1016/0094-5765(77)90096-0	journal	November 1977
Optimal prediction with memory Chorin, Alexandre J.; Hald, Ole H.; Kupferman, Raz Physica D: Nonlinear Phenomena, Vol. 166, Issue 3-4 https://doi.org/10.1016/S0167-2789(02)00446-3	journal	June 2002
A dynamic subgrid scale model for Large Eddy Simulations based on the Mori–Zwanzig formalism Parish, Eric J.; Duraisamy, Karthik Journal of Computational Physics, Vol. 349 https://doi.org/10.1016/j.jcp.2017.07.053	journal	November 2017
Data-driven model reduction, Wiener projections, and the Koopman-Mori-Zwanzig formalism Lin, Kevin K.; Lu, Fei Journal of Computational Physics, Vol. 424 https://doi.org/10.1016/j.jcp.2020.109864	journal	January 2021
Machine learning for prediction with missing dynamics Harlim, John; Jiang, Shixiao W.; Liang, Senwei Journal of Computational Physics, Vol. 428 https://doi.org/10.1016/j.jcp.2020.109922	journal	March 2021
Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems Qian, Elizabeth; Kramer, Boris; Peherstorfer, Benjamin Physica D: Nonlinear Phenomena, Vol. 406 https://doi.org/10.1016/j.physd.2020.132401	journal	May 2020
Kernel-based prediction of non-Markovian time series Gilani, Faheem; Giannakis, Dimitrios; Harlim, John Physica D: Nonlinear Phenomena, Vol. 418 https://doi.org/10.1016/j.physd.2020.132829	journal	April 2021
Mathematics for Machine Learning Deisenroth, Marc Peter; Faisal, A. Aldo; Ong, Cheng Soon https://doi.org/10.1017/9781108679930	book	January 2020
Non-linear dynamics and statistical theories for basic geophysical flows Majda, Andrew; Wang, Xiaoming https://doi.org/10.1017/CBO9780511616778	book	January 2006
Spectral analysis of nonlinear flows Rowley, Clarence W.; MeziĆ, Igor; Bagheri, Shervin Journal of Fluid Mechanics, Vol. 641 https://doi.org/10.1017/S0022112009992059	journal	November 2009
Dynamic mode decomposition of numerical and experimental data Schmid, Peter J. Journal of Fluid Mechanics, Vol. 656 https://doi.org/10.1017/S0022112010001217	journal	July 2010
Lyapunov Exponents of the Kuramoto–Sivashinsky pde Edson, Russell A.; Bunder, J. E.; Mattner, Trent W. The ANZIAM Journal, Vol. 61, Issue 3 https://doi.org/10.1017/S1446181119000105	journal	July 2019
Deep learning for universal linear embeddings of nonlinear dynamics Lusch, Bethany; Kutz, J. Nathan; Brunton, Steven L. Nature Communications, Vol. 9, Issue 1 https://doi.org/10.1038/s41467-018-07210-0	journal	November 2018
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators Lu, Lu; Jin, Pengzhan; Pang, Guofei Nature Machine Intelligence, Vol. 3, Issue 3 https://doi.org/10.1038/s42256-021-00302-5	journal	March 2021
Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator Li, Qianxiao; Dietrich, Felix; Bollt, Erik M. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 27, Issue 10 https://doi.org/10.1063/1.4993854	journal	October 2017
Time-lagged autoencoders: Deep learning of slow collective variables for molecular kinetics Wehmeyer, Christoph; Noé, Frank The Journal of Chemical Physics, Vol. 148, Issue 24 https://doi.org/10.1063/1.5011399	journal	June 2018
On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD Bollt, Erik Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, Issue 1 https://doi.org/10.1063/5.0024890	journal	January 2021
Data-driven learning of Mori–Zwanzig operators for isotropic turbulence Tian, Yifeng; Lin, Yen Ting; Anghel, Marian Physics of Fluids, Vol. 33, Issue 12 https://doi.org/10.1063/5.0070548	journal	December 2021
Computing generalized Langevin equations and generalized Fokker-Planck equations Darve, E.; Solomon, J.; Kia, A. Proceedings of the National Academy of Sciences, Vol. 106, Issue 27 https://doi.org/10.1073/pnas.0902633106	journal	June 2009
Discrete approach to stochastic parametrization and dimension reduction in nonlinear dynamics Chorin, Alexandre J.; Lu, Fei Proceedings of the National Academy of Sciences, Vol. 112, Issue 32 https://doi.org/10.1073/pnas.1512080112	journal	July 2015
Discovering governing equations from data by sparse identification of nonlinear dynamical systems Brunton, Steven L.; Proctor, Joshua L.; Kutz, J. Nathan Proceedings of the National Academy of Sciences, Vol. 113, Issue 15 https://doi.org/10.1073/pnas.1517384113	journal	March 2016
Dynamical Systems of Continuous Spectra Koopman, B. O.; Neumann, J. v. Proceedings of the National Academy of Sciences, Vol. 18, Issue 3 https://doi.org/10.1073/pnas.18.3.255	journal	March 1932
Optimal renormalization of multiscale systems Price, Jacob; Meuris, Brek; Shapiro, Madelyn Proceedings of the National Academy of Sciences, Vol. 118, Issue 37 https://doi.org/10.1073/pnas.2102266118	journal	September 2021
Reproducing Kernel Hilbert Spaces for Penalized Regression: A Tutorial Nosedal-Sanchez, Alvaro; Storlie, Curtis B.; Lee, Thomas C. M. The American Statistician, Vol. 66, Issue 1 https://doi.org/10.1080/00031305.2012.678196	journal	February 2012
Nonequilibrium Statistical Mechanics Zwanzig, Robert Oxford Scholarship Online https://doi.org/10.1093/oso/9780195140187.001.0001	book	April 2001
Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control Fasel, U.; Kutz, J. N.; Brunton, B. W. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 478, Issue 2260 https://doi.org/10.1098/rspa.2021.0904	journal	April 2022
Validity of the essential assumption in a projection operator method Okamura, Makoto Physical Review E, Vol. 74, Issue 4 https://doi.org/10.1103/PhysRevE.74.046210	journal	October 2006
Dynamic structures of the time correlation functions of chaotic nonequilibrium fluctuations Mori, Hazime; Okamura, Makoto Physical Review E, Vol. 76, Issue 6 https://doi.org/10.1103/PhysRevE.76.061104	journal	December 2007
Non-Markovian closure models for large eddy simulations using the Mori-Zwanzig formalism Parish, Eric J.; Duraisamy, Karthik Physical Review Fluids, Vol. 2, Issue 1 https://doi.org/10.1103/PhysRevFluids.2.014604	journal	January 2017
Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems No authors listed IEEE Transactions on Neural Networks, Vol. 6, Issue 4 https://doi.org/10.1109/72.392253	journal	July 1995
On Flame Propagation Under Conditions of Stoichiometry Sivashinsky, G. I. SIAM Journal on Applied Mathematics, Vol. 39, Issue 1 https://doi.org/10.1137/0139007	journal	August 1980
Ergodic Theory, Dynamic Mode Decomposition, and Computation of Spectral Properties of the Koopman Operator Arbabi, Hassan; Mezić, Igor SIAM Journal on Applied Dynamical Systems, Vol. 16, Issue 4 https://doi.org/10.1137/17M1125236	journal	January 2017
Renormalized Reduced Order Models with Memory for Long Time Prediction Price, Jacob; Stinis, Panos Multiscale Modeling & Simulation, Vol. 17, Issue 1 https://doi.org/10.1137/17M1151389	journal	January 2019
Data-Driven Learning for the Mori--Zwanzig Formalism: A Generalization of the Koopman Learning Framework Lin, Yen Ting; Tian, Yifeng; Livescu, Daniel SIAM Journal on Applied Dynamical Systems, Vol. 20, Issue 4 https://doi.org/10.1137/21M1401759	journal	January 2021
Fourth-Order Time-Stepping for Stiff PDEs Kassam, Aly-Khan; Trefethen, Lloyd N. SIAM Journal on Scientific Computing, Vol. 26, Issue 4 https://doi.org/10.1137/S1064827502410633	journal	January 2005
Transport, Collective Motion, and Brownian Motion Mori, Hazime Progress of Theoretical Physics, Vol. 33, Issue 3 https://doi.org/10.1143/PTP.33.423	journal	March 1965
Diffusion-Induced Chaos in Reaction Systems Kuramoto, Yoshiki Progress of Theoretical Physics Supplement, Vol. 64 https://doi.org/10.1143/PTPS.64.346	journal	January 1978
Some Recent Developments in Turbulence Closure Modeling Durbin, Paul A. Annual Review of Fluid Mechanics, Vol. 50, Issue 1 https://doi.org/10.1146/annurev-fluid-122316-045020	journal	January 2018
Long Short-Term Memory Hochreiter, Sepp; Schmidhuber, Jürgen Neural Computation, Vol. 9, Issue 8 https://doi.org/10.1162/neco.1997.9.8.1735	journal	November 1997
Deterministic Nonperiodic Flow Lorenz, Edward N. Journal of the Atmospheric Sciences, Vol. 20, Issue 2 https://doi.org/10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2	journal	March 1963
Non-Markovian out-of-equilibrium dynamics: A general numerical procedure to construct time-dependent memory kernels for coarse-grained observables Meyer, Hugues; Pelagejcev, Philipp; Schilling, Tanja EPL (Europhysics Letters), Vol. 128, Issue 4 https://doi.org/10.1209/0295-5075/128/40001	journal	November 2019
Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems Yeung, Enoch; Kundu, Soumya; Hodas, Nathan 2019 American Control Conference (ACC) https://doi.org/10.23919/ACC.2019.8815339	conference	July 2019
Extracting and Modeling the Effects of Small-Scale Fluctuations on Large-Scale Fluctuations by Mori–Zwanzig Projection Operator Method Maeyama, Shinya; Watanabe, Tomo-Hiko Journal of the Physical Society of Japan, Vol. 89, Issue 2 https://doi.org/10.7566/JPSJ.89.024401	journal	February 2020

Similar Records

Data-Driven Learning for the Mori--Zwanzig Formalism: A Generalization of the Koopman Learning Framework

Journal Article · Wed Dec 15 19:00:00 EST 2021 · SIAM Journal on Applied Dynamical Systems · OSTI ID:1872339

Data-driven learning of Mori–Zwanzig operators for isotropic turbulence

Journal Article · Wed Dec 08 19:00:00 EST 2021 · Physics of Fluids · OSTI ID:2565539

Related Subjects

97 MATHEMATICS AND COMPUTING
Koopman representation
Mathematics
Mori–Zwanzig formalism
data-driven learning
neural networks
nonlinear projection operators
regression

Regression-Based Projection for Learning Mori–Zwanzig Operators

Citation Formats

References (50)

Similar Records

Related Subjects