A multifidelity approach to continual learning for physical systems

Howard, Amanda; Fu, Yucheng; Stinis, Panos

doi:10.1088/2632-2153/ad45b2

Title: A multifidelity approach to continual learning for physical systems

Journal Article · 16 May 2024 · Machine Learning: Science and Technology

DOI: https://doi.org/10.1088/2632-2153/ad45b2 · OSTI ID:2350972

;

Abstract We introduce a novel continual learning method based on multifidelity deep neural networks. This method learns the correlation between the output of previously trained models and the desired output of the model on the current training dataset, limiting catastrophic forgetting. On its own the multifidelity continual learning method shows robust results that limit forgetting across several datasets. Additionally, we show that the multifidelity method can be combined with existing continual learning methods, including replay and memory aware synapses, to further limit catastrophic forgetting. The proposed continual learning method is especially suited for physical problems where the data satisfy the same physical laws on each domain, or for physics-informed neural networks, because in these cases we expect there to be a strong correlation between the output of the previous model and the model on the current training domain.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2350972

Journal Information:: Machine Learning: Science and Technology, Journal Name: Machine Learning: Science and Technology Journal Issue: 2 Vol. 5; ISSN 2632-2153

Publisher:: IOP PublishingCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (49)

Recent Progress in Redox Flow Battery Research and Development Wang, Wei; Luo, Qingtao; Li, Bin Advanced Functional Materials, Vol. 23, Issue 8, p. 970-986 https://doi.org/10.1002/adfm.201200694	journal	September 2012
DAE-PINN: a physics-informed neural network model for simulating differential algebraic equations with application to power networks Moya, Christian; Lin, Guang Neural Computing and Applications, Vol. 35, Issue 5 https://doi.org/10.1007/s00521-022-07886-y	journal	October 2022
Applying machine learning to study fluid mechanics Brunton, Steven L. Acta Mechanica Sinica, Vol. 37, Issue 12 https://doi.org/10.1007/s10409-021-01143-6	journal	December 2021
Physics-informed neural networks (PINNs) for fluid mechanics: a review Cai, Shengze; Mao, Zhiping; Wang, Zhicheng Acta Mechanica Sinica, Vol. 37, Issue 12 https://doi.org/10.1007/s10409-021-01148-1	journal	December 2021
Redox flow batteries a review Weber, Adam Z.; Mench, Matthew M.; Meyers, Jeremy P. Journal of Applied Electrochemistry, Vol. 41, Issue 10, p. 1137-1164 https://doi.org/10.1007/s10800-011-0348-2	journal	September 2011
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next Cuomo, Salvatore; Di Cola, Vincenzo Schiano; Giampaolo, Fabio Journal of Scientific Computing, Vol. 92, Issue 3 https://doi.org/10.1007/s10915-022-01939-z	journal	July 2022
Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting Hou, Jie; Li, Ying; Ying, Shihui Nonlinear Dynamics, Vol. 111, Issue 16 https://doi.org/10.1007/s11071-023-08654-w	journal	July 2023
A study of the V(II)/V(III) redox couple for redox flow cell applications Sum, E.; Skyllas-Kazacos, M. Journal of Power Sources, Vol. 15, Issue 2-3, p. 179-190 https://doi.org/10.1016/0378-7753(85)80071-9	journal	June 1985
Predicting city-scale daily electricity consumption using data-driven models Wang, Zhe; Hong, Tianzhen; Li, Han Advances in Applied Energy, Vol. 2 https://doi.org/10.1016/j.adapen.2021.100025	journal	May 2021
Physics-informed neural networks for high-speed flows Mao, Zhiping; Jagtap, Ameya D.; Karniadakis, George Em Computer Methods in Applied Mechanics and Engineering, Vol. 360 https://doi.org/10.1016/j.cma.2019.112789	journal	March 2020
PPINN: Parareal physics-informed neural network for time-dependent PDEs Meng, Xuhui; Li, Zhen; Zhang, Dongkun Computer Methods in Applied Mechanics and Engineering, Vol. 370 https://doi.org/10.1016/j.cma.2020.113250	journal	October 2020
Multi-fidelity deep neural network surrogate model for aerodynamic shape optimization Zhang, Xinshuai; Xie, Fangfang; Ji, Tingwei Computer Methods in Applied Mechanics and Engineering, Vol. 373 https://doi.org/10.1016/j.cma.2020.113485	journal	January 2021
A novel sequential method to train physics informed neural networks for Allen Cahn and Cahn Hilliard equations Mattey, Revanth; Ghosh, Susanta Computer Methods in Applied Mechanics and Engineering, Vol. 390 https://doi.org/10.1016/j.cma.2021.114474	journal	February 2022
Galerkin neural network approximation of singularly-perturbed elliptic systems Ainsworth, Mark; Dong, Justin Computer Methods in Applied Mechanics and Engineering, Vol. 402 https://doi.org/10.1016/j.cma.2022.115169	journal	December 2022
A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks Wu, Chenxi; Zhu, Min; Tan, Qinyang Computer Methods in Applied Mechanics and Engineering, Vol. 403 https://doi.org/10.1016/j.cma.2022.115671	journal	January 2023
A multifidelity deep operator network approach to closure for multiscale systems Ahmed, Shady E.; Stinis, Panos Computer Methods in Applied Mechanics and Engineering, Vol. 414 https://doi.org/10.1016/j.cma.2023.116161	journal	September 2023
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems Meng, Xuhui; Karniadakis, George Em Journal of Computational Physics, Vol. 401 https://doi.org/10.1016/j.jcp.2019.109020	journal	January 2020
NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations Jin, Xiaowei; Cai, Shengze; Li, Hui Journal of Computational Physics, Vol. 426 https://doi.org/10.1016/j.jcp.2020.109951	journal	February 2021
When and why PINNs fail to train: A neural tangent kernel perspective Wang, Sifan; Yu, Xinling; Perdikaris, Paris Journal of Computational Physics, Vol. 449 https://doi.org/10.1016/j.jcp.2021.110768	journal	January 2022
Long-time integration of parametric evolution equations with physics-informed DeepONets Wang, Sifan; Perdikaris, Paris Journal of Computational Physics, Vol. 475 https://doi.org/10.1016/j.jcp.2022.111855	journal	February 2023
Multifidelity deep operator networks for data-driven and physics-informed problems Howard, Amanda A.; Perego, Mauro; Karniadakis, George Em Journal of Computational Physics, Vol. 493 https://doi.org/10.1016/j.jcp.2023.112462	journal	November 2023
Analytical modeling for redox flow battery design Chen, Yunxiang; Xu, Zhijie; Wang, Chao Journal of Power Sources, Vol. 482 https://doi.org/10.1016/j.jpowsour.2020.228817	journal	January 2021
A two-dimensional analytical unit cell model for redox flow battery evaluation and optimization Chen, Yunxiang; Bao, Jie; Xu, Zhijie Journal of Power Sources, Vol. 506 https://doi.org/10.1016/j.jpowsour.2021.230192	journal	September 2021
Continual lifelong learning with neural networks: A review Parisi, German I.; Kemker, Ronald; Part, Jose L. Neural Networks, Vol. 113 https://doi.org/10.1016/j.neunet.2019.01.012	journal	May 2019
Towards multi-fidelity deep learning of wind turbine wakes Pawar, Suraj; Sharma, Ashesh; Vijayakumar, Ganesh Renewable Energy, Vol. 200 https://doi.org/10.1016/j.renene.2022.10.013	journal	November 2022
Vanadium Flow Battery for Energy Storage: Prospects and Challenges Ding, Cong; Zhang, Huamin; Li, Xianfeng The Journal of Physical Chemistry Letters, Vol. 4, Issue 8 https://doi.org/10.1021/jz4001032	journal	March 2013
Physics‐Informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions Rasht‐Behesht, Majid; Huber, Christian; Shukla, Khemraj Journal of Geophysical Research: Solid Earth, Vol. 127, Issue 5 https://doi.org/10.1029/2021JB023120	journal	April 2022
Physics-informed machine learning Karniadakis, George Em; Kevrekidis, Ioannis G.; Lu, Lu Nature Reviews Physics, Vol. 3, Issue 6 https://doi.org/10.1038/s42254-021-00314-5	journal	May 2021
Overcoming catastrophic forgetting in neural networks Kirkpatrick, James; Pascanu, Razvan; Rabinowitz, Neil Proceedings of the National Academy of Sciences, Vol. 114, Issue 13 https://doi.org/10.1073/pnas.1611835114	journal	March 2017
Recipes for when physics fails: recovering robust learning of physics informed neural networks Bajaj, Chandrajit; McLennan, Luke; Andeen, Timothy Machine Learning: Science and Technology, Vol. 4, Issue 1 https://doi.org/10.1088/2632-2153/acb416	journal	February 2023
Machine learning of hidden variables in multiscale fluid simulation Joglekar, Archis S.; Thomas, Alexander G. R. Machine Learning: Science and Technology, Vol. 4, Issue 3 https://doi.org/10.1088/2632-2153/acf81a	journal	September 2023
Stokesian processes : inferring Stokes flows using physics-informed Gaussian processes Molina, John J.; Ogawa, Kenta; Taniguchi, Takashi Machine Learning: Science and Technology, Vol. 4, Issue 4 https://doi.org/10.1088/2632-2153/ad0286	journal	October 2023
Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport Lu, Lu; Pestourie, Raphaël; Johnson, Steven G. Physical Review Research, Vol. 4, Issue 2 https://doi.org/10.1103/PhysRevResearch.4.023210	journal	June 2022
Deep Physical Informed Neural Networks for Metamaterial Design Fang, Zhiwei; Zhan, Justin IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2019.2963375	journal	January 2020
A continual learning survey: Defying forgetting in classification tasks Delange, Matthias; Aljundi, Rahaf; Masana, Marc IEEE Transactions on Pattern Analysis and Machine Intelligence https://doi.org/10.1109/TPAMI.2021.3057446	journal	January 2021
Applications of Physics-Informed Neural Networks in Power Systems - A Review Huang, Bin; Wang, Jianhui IEEE Transactions on Power Systems, Vol. 38, Issue 1 https://doi.org/10.1109/TPWRS.2022.3162473	journal	January 2023
Physics-Guided Neural Network for Load Margin Assessment of Power Systems Bento, Murilo E. C. IEEE Transactions on Power Systems, Vol. 39, Issue 1 https://doi.org/10.1109/TPWRS.2023.3266236	journal	January 2024
Efficient training of physics‐informed neural networks via importance sampling Nabian, Mohammad Amin; Gladstone, Rini Jasmine; Meidani, Hadi Computer-Aided Civil and Infrastructure Engineering, Vol. 36, Issue 8 https://doi.org/10.1111/mice.12685	journal	April 2021
Multi-Fidelity Physics-Constrained Neural Network and Its Application in Materials Modeling Liu, Dehao; Wang, Yan Journal of Mechanical Design, Vol. 141, Issue 12 https://doi.org/10.1115/1.4044400	journal	September 2019
Data-driven physics-informed constitutive metamodeling of complex fluids: A multifidelity neural network (MFNN) framework Mahmoudabadbozchelou, Mohammadamin; Caggioni, Marco; Shahsavari, Setareh Journal of Rheology, Vol. 65, Issue 2 https://doi.org/10.1122/8.0000138	journal	March 2021
Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations Raissi, Maziar; Yazdani, Alireza; Karniadakis, George Em Science, Vol. 367, Issue 6481 https://doi.org/10.1126/science.aaw4741	journal	January 2020
Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control Ainsworth, Mark; Dong, Justin SIAM Journal on Scientific Computing, Vol. 43, Issue 4 https://doi.org/10.1137/20M1366587	journal	January 2021
Failure-Informed Adaptive Sampling for PINNs Gao, Zhiwei; Yan, Liang; Zhou, Tao SIAM Journal on Scientific Computing, Vol. 45, Issue 4 https://doi.org/10.1137/22M1527763	journal	July 2023
Physics-informed neural networks for inverse problems in nano-optics and metamaterials Chen, Yuyao; Lu, Lu; Karniadakis, George Em Optics Express, Vol. 28, Issue 8 https://doi.org/10.1364/OE.384875	journal	January 2020
AdapterHub: A Framework for Adapting Transformers Pfeiffer, Jonas; Rücklé, Andreas; Poth, Clifton Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations https://doi.org/10.18653/v1/2020.emnlp-demos.7	conference	January 2020
Multi-Stage Neural Networks: Function Approximator of Machine Precision Wang, Yongji; Lai, Ching-Yao https://doi.org/10.2139/ssrn.4523251	preprint	July 2023
Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence Baker, Nathan; Alexander, Frank; Bremer, Timo https://doi.org/10.2172/1478744	report	February 2019
Extended Physics-Informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition Based Deep Learning Framework for Nonlinear Partial Differential Equations Karniadakis, Ameya D. Jagtap & George Em Communications in Computational Physics, Vol. 28, Issue 5 https://doi.org/10.4208/cicp.oa-2020-0164	journal	June 2020

Similar Records

Mitigating Catastrophic Forgetting in Deep Learning in a Streaming Setting Using Historical Summary

Conference · 2021 · OSTI ID:1885239

Dash, Sajal; Yin, Junqi; Shankar, Mallikarjun (Arjun); +2 more

Comparison of multifidelity machine learning models for potential energy surfaces

Journal Article · 2023 · Journal of Chemical Physics · OSTI ID:2420722

Goodlett, Stephen M.; Turney, Justin M.; Schaefer, Henry F.

Noise-Based Selection of Robust Inherited Model for Accurate Continual Learning

Conference · 2020 · OSTI ID:1665964

Du, Xiaocong; Li, Zheng; Seo, Jae-Sun; +2 more

Title: A multifidelity approach to continual learning for physical systems

Citation Formats

References (49)

Similar Records

Related Subjects