On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Wang, Sifan; Wang, Hanwen; Perdikaris, Paris

doi:10.1016/j.cma.2021.113938

On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Journal Article · Wed Jun 02 00:00:00 EDT 2021 · Computer Methods in Applied Mechanics and Engineering

DOI:https://doi.org/10.1016/j.cma.2021.113938· OSTI ID:1976965

Wang, Sifan ^[1]; ^[2]; ^[2]

University of Pennsylvania, Philadelphia, PA (United States); OSTI
University of Pennsylvania, Philadelphia, PA (United States)

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. Here in this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction–diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior.

View Accepted Manuscript (DOE)

Research Organization:: Raytheon Technologies Corporation, Waltham, MA (United States); University of Pennsylvania, Philadelphia, PA (United States)

Sponsoring Organization:: Air Force Office of Scientific Research (AFOSR); USDOE Advanced Research Projects Agency - Energy (ARPA-E)

Grant/Contract Number:: AR0001201; SC0019116

OSTI ID:: 1976965

Alternate ID(s):: OSTI ID: 1786215

Journal Information:: Computer Methods in Applied Mechanics and Engineering, Journal Name: Computer Methods in Applied Mechanics and Engineering Journal Issue: C Vol. 384; ISSN 0045-7825

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (25)

A hybrid neural network-first principles approach to process modeling Psichogios, Dimitris C.; Ungar, Lyle H. AIChE Journal, Vol. 38, Issue 10 https://doi.org/10.1002/aic.690381003	journal	October 1992
Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations Sivashinsky, G. I. Acta Astronautica, Vol. 4, Issue 11-12 https://doi.org/10.1016/0094-5765(77)90096-0	journal	November 1977
PhyGeoNet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain Gao, Han; Sun, Luning; Wang, Jian-Xun Journal of Computational Physics, Vol. 428 https://doi.org/10.1016/j.jcp.2020.110079	journal	March 2021
Wide neural networks of any depth evolve as linear models under gradient descent ^* Lee, Jaehoon; Xiao, Lechao; Schoenholz, Samuel S. Journal of Statistical Mechanics: Theory and Experiment, Vol. 2020, Issue 12 https://doi.org/10.1088/1742-5468/abc62b	journal	December 2020
Diffusion-Induced Chaos in Reaction Systems Kuramoto, Yoshiki Progress of Theoretical Physics Supplement, Vol. 64 https://doi.org/10.1143/ptps.64.346	journal	January 1978
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios arXiv https://doi.org/10.48550/arxiv.1901.06314	text	January 2019
On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs Shin, Yeonjong; Darbon, Jerome; Karniadakis, George Em arXiv https://doi.org/10.48550/arxiv.2004.01806	text	January 2020
Exponential Time Differencing for Stiff Systems Cox, S. M.; Matthews, P. C. Journal of Computational Physics, Vol. 176, Issue 2 https://doi.org/10.1006/jcph.2002.6995	journal	March 2002
Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks Kissas, Georgios; Yang, Yibo; Hwuang, Eileen Computer Methods in Applied Mechanics and Engineering, Vol. 358 https://doi.org/10.1016/j.cma.2019.112623	journal	January 2020
Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data Sun, Luning; Gao, Han; Pan, Shaowu Computer Methods in Applied Mechanics and Engineering, Vol. 361 https://doi.org/10.1016/j.cma.2019.112732	journal	April 2020
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.024	journal	October 2019
Adversarial uncertainty quantification in physics-informed neural networks Yang, Yibo; Perdikaris, Paris Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.027	journal	October 2019
Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks Karumuri, Sharmila; Tripathy, Rohit; Bilionis, Ilias Journal of Computational Physics, Vol. 404 https://doi.org/10.1016/j.jcp.2019.109120	journal	March 2020
Array programming with NumPy Harris, Charles R.; Millman, K. Jarrod; van der Walt, Stéfan J. Nature, Vol. 585, Issue 7825 https://doi.org/10.1038/s41586-020-2649-2	journal	September 2020
Solving high-dimensional partial differential equations using deep learning Han, Jiequn; Jentzen, Arnulf; E., Weinan Proceedings of the National Academy of Sciences, Vol. 115, Issue 34 https://doi.org/10.1073/pnas.1718942115	journal	August 2018
Bayesian differential programming for robust systems identification under uncertainty Yang, Yibo; Aziz Bhouri, Mohamed; Perdikaris, Paris Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 476, Issue 2243 https://doi.org/10.1098/rspa.2020.0290	journal	November 2020
Artificial neural networks for solving ordinary and partial differential equations Lagaris, I. E.; Likas, A.; Fotiadis, D. I. IEEE Transactions on Neural Networks, Vol. 9, Issue 5 https://doi.org/10.1109/72.712178	journal	January 1998
Deep Physical Informed Neural Networks for Metamaterial Design Fang, Zhiwei; Zhan, Justin IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2019.2963375	journal	January 2020
On the Eigenspectrum of the Gram Matrix and the Generalization Error of Kernel-PCA Shawe-Taylor, J.; Williams, C. K. I.; Cristianini, N. IEEE Transactions on Information Theory, Vol. 51, Issue 7 https://doi.org/10.1109/TIT.2005.850052	journal	July 2005
Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations Raissi, Maziar; Yazdani, Alireza; Karniadakis, George Em Science, Vol. 367, Issue 6481 https://doi.org/10.1126/science.aaw4741	journal	January 2020
Diffusion-Induced Chaos in Reaction Systems Kuramoto, Yoshiki Progress of Theoretical Physics Supplement, Vol. 64 https://doi.org/10.1143/PTPS.64.346	journal	January 1978
Physics-informed neural networks for inverse problems in nano-optics and metamaterials Chen, Yuyao; Lu, Lu; Karniadakis, George Em Optics Express, Vol. 28, Issue 8 https://doi.org/10.1364/OE.384875	journal	January 2020
Systems biology informed deep learning for inferring parameters and hidden dynamics Yazdani, Alireza; Lu, Lu; Raissi, Maziar PLOS Computational Biology, Vol. 16, Issue 11 https://doi.org/10.1371/journal.pcbi.1007575	journal	November 2020
Physics-Informed Neural Networks for Cardiac Activation Mapping Sahli Costabal, Francisco; Yang, Yibo; Perdikaris, Paris Frontiers in Physics, Vol. 8 https://doi.org/10.3389/fphy.2020.00042	journal	February 2020

Cited By (2)

A deep neural network approach on solving the linear transport model under diffusive scaling Liu, Liu; Zeng, Tieyong; Zhang, Zecheng arXiv https://doi.org/10.48550/arxiv.2102.12408	preprint	January 2021
Solving and Learning Nonlinear PDEs with Gaussian Processes Chen, Yifan; Hosseini, Bamdad; Owhadi, Houman arXiv https://doi.org/10.48550/arxiv.2103.12959	preprint	January 2021

Similar Records

When and why PINNs fail to train: A neural tangent kernel perspective

Journal Article · Sun Oct 10 20:00:00 EDT 2021 · Journal of Computational Physics · OSTI ID:1977272

Respecting causality for training physics-informed neural networks

Journal Article · Tue Feb 06 19:00:00 EST 2024 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:2339526

B-PINNs: Bayesian Physics-informal Neural Networks for Forward and Inverse PDE Problems with Noisy Data

Journal Article · Thu Jan 14 23:00:00 EST 2021 · Journal of Computational Physics · OSTI ID:1763962

Related Subjects

42 ENGINEERING
97 MATHEMATICS AND COMPUTING
deep learning
neural tangent kernel
partial differential equations
scientific machine learning
spectral bias

On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

Citation Formats

References (25)

Cited By (2)

Similar Records

Related Subjects