Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Lu, Lu; Jin, Pengzhan; Pang, Guofei; Zhang, Zhongqiang; Karniadakis, George Em

doi:10.1038/s42256-021-00302-5

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Journal Article · Thu Mar 18 00:00:00 EDT 2021 · Nature Machine Intelligence

DOI:https://doi.org/10.1038/s42256-021-00302-5· OSTI ID:2281727

^[1]; ^[2]; Pang, Guofei ^[3]; ^[4]; ^[3]

Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States); Brown University
Brown Univ., Providence, RI (United States); Chinese Academy of Sciences (CAS), Beijing (China)
Brown Univ., Providence, RI (United States)
Worcester Polytechnic Institute, MA (United States)

It is widely known that neural networks (NNs) are universal approximators of continuous functions. However, a less known but powerful result is that a NN with a single hidden layer can accurately approximate any nonlinear continuous operator. This universal approximation theorem of operators is suggestive of the structure and potential of deep neural networks (DNNs) in learning continuous operators or complex systems from streams of scattered data. Here, in this work, we thus extend this theorem to DNNs. We design a new network with small generalization error, the deep operator network (DeepONet), which consists of a DNN for encoding the discrete input function space (branch net) and another DNN for encoding the domain of the output functions (trunk net). We demonstrate that DeepONet can learn various explicit operators, such as integrals and fractional Laplacians, as well as implicit operators that represent deterministic and stochastic differential equations. We study different formulations of the input function space and its effect on the generalization error for 16 different diverse applications.

View Accepted Manuscript (DOE)

Research Organization:: Brown Univ., Providence, RI (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: SC0019453

OSTI ID:: 2281727

Alternate ID(s):: OSTI ID: 1853304

Journal Information:: Nature Machine Intelligence, Journal Name: Nature Machine Intelligence Journal Issue: 3 Vol. 3; ISSN 2522-5839

Publisher:: Springer NatureCopyright Statement

Country of Publication:: United States

Language:: English

References (29)

A hybrid neural network-first principles approach to process modeling Psichogios, Dimitris C.; Ungar, Lyle H. AIChE Journal, Vol. 38, Issue 10 https://doi.org/10.1002/aic.690381003	journal	October 1992
On a Formula for the L2 Wasserstein Metric between Measures on Euclidean and Hilbert Spaces Gelbrich, Matthias Mathematische Nachrichten, Vol. 147, Issue 1 https://doi.org/10.1002/mana.19901470121	journal	January 1990
Approximation by superpositions of a sigmoidal function Cybenko, G. Mathematics of Control, Signals, and Systems, Vol. 2, Issue 4 https://doi.org/10.1007/BF02551274	journal	December 1989
Multilayer feedforward networks are universal approximators Hornik, Kurt; Stinchcombe, Maxwell; White, Halbert Neural Networks, Vol. 2, Issue 5 https://doi.org/10.1016/0893-6080(89)90020-8	journal	January 1989
Identification of distributed parameter systems: A neural net based approach González-García, R.; Rico-Martínez, R.; Kevrekidis, I. G. Computers & Chemical Engineering, Vol. 22 https://doi.org/10.1016/S0098-1354(98)00191-4	journal	March 1998
Fractional Sturm–Liouville eigen-problems: Theory and numerical approximation Zayernouri, Mohsen; Karniadakis, George Em Journal of Computational Physics, Vol. 252 https://doi.org/10.1016/j.jcp.2013.06.031	journal	November 2013
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.024	journal	October 2019
ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains Winovich, Nick; Ramani, Karthik; Lin, Guang Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.026	journal	October 2019
Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems Zhang, Dongkun; Lu, Lu; Guo, Ling Journal of Computational Physics, Vol. 397 https://doi.org/10.1016/j.jcp.2019.07.048	journal	November 2019
What is the fractional Laplacian? A comparative review with new results Lischke, Anna; Pang, Guofei; Gulian, Mamikon Journal of Computational Physics, Vol. 404 https://doi.org/10.1016/j.jcp.2019.109009	journal	March 2020
DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks Cai, Shengze; Wang, Zhicheng; Lu, Lu Journal of Computational Physics, Vol. 436 https://doi.org/10.1016/j.jcp.2021.110296	journal	July 2021
Functional multi-layer perceptron: a non-linear tool for functional data analysis Rossi, Fabrice; Conan-Guez, Brieuc Neural Networks, Vol. 18, Issue 1 https://doi.org/10.1016/j.neunet.2004.07.001	journal	January 2005
Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness Jin, Pengzhan; Lu, Lu; Tang, Yifa Neural Networks, Vol. 130 https://doi.org/10.1016/j.neunet.2020.06.024	journal	October 2020
DISCRETE- vs. CONTINUOUS-TIME NONLINEAR SIGNAL PROCESSING OF Cu ELECTRODISSOLUTION DATA Rico-MartÍNez, R.; Krischer, K.; Kevrekidis, I. G. Chemical Engineering Communications, Vol. 118, Issue 1 https://doi.org/10.1080/00986449208936084	journal	November 1992
Approximations of continuous functionals by neural networks with application to dynamic systems Chen, T.; Chen, H. IEEE Transactions on Neural Networks, Vol. 4, Issue 6 https://doi.org/10.1109/72.286886	journal	January 1993
Approximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function neural networks No authors listed IEEE Transactions on Neural Networks, Vol. 6, Issue 4 https://doi.org/10.1109/72.392252	journal	July 1995
Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems No authors listed IEEE Transactions on Neural Networks, Vol. 6, Issue 4 https://doi.org/10.1109/72.392253	journal	July 1995
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Data-driven discovery of partial differential equations Rudy, Samuel H.; Brunton, Steven L.; Proctor, Joshua L. Science Advances, Vol. 3, Issue 4 https://doi.org/10.1126/sciadv.1602614	journal	April 2017
fPINNs: Fractional Physics-Informed Neural Networks Pang, Guofei; Lu, Lu; Karniadakis, George Em SIAM Journal on Scientific Computing, Vol. 41, Issue 4 https://doi.org/10.1137/18M1229845	journal	January 2019
DeepXDE: A Deep Learning Library for Solving Differential Equations Lu, Lu; Meng, Xuhui; Mao, Zhiping SIAM Review, Vol. 63, Issue 1 https://doi.org/10.1137/19M1274067	journal	January 2021
Neural Networks for Functional Approximation and System Identification Mhaskar, H. N.; Hahm, Nahmwoo Neural Computation, Vol. 9, Issue 1 https://doi.org/10.1162/neco.1997.9.1.143	journal	January 1997
Physics-informed neural networks for inverse problems in nano-optics and metamaterials Chen, Yuyao; Lu, Lu; Karniadakis, George Em Optics Express, Vol. 28, Issue 8 https://doi.org/10.1364/OE.384875	journal	January 2020
Systems biology informed deep learning for inferring parameters and hidden dynamics Yazdani, Alireza; Lu, Lu; Raissi, Maziar PLOS Computational Biology, Vol. 16, Issue 11 https://doi.org/10.1371/journal.pcbi.1007575	journal	November 2020
Massive Exploration of Neural Machine Translation Architectures Britz, Denny; Goldie, Anna; Luong, Minh-Thang Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing https://doi.org/10.18653/v1/D17-1151	conference	January 2017
GMLS-Nets: A Framework for Learning from Unstructured Data Trask, Nathaniel; Patel, Ravi; Gross, Ben https://doi.org/10.2172/1763256	report	September 2019
Feature-wise transformations Dumoulin, Vincent; Perez, Ethan; Schucher, Nathan Distill, Vol. 3, Issue 7 https://doi.org/10.23915/distill.00011	journal	July 2018
Dying ReLU and Initialization: Theory and Numerical Examples Lu, Lu Communications in Computational Physics, Vol. 28, Issue 5 https://doi.org/10.4208/cicp.OA-2020-0165	journal	June 2020
Equation-Free, Coarse-Grained Multiscale Computation: Enabling Mocroscopic Simulators to Perform System-Level Analysis Gear, C. William; Hyman, James M.; Kevrekidid, Panagiotis G. Communications in Mathematical Sciences, Vol. 1, Issue 4 https://doi.org/10.4310/CMS.2003.v1.n4.a5	journal	January 2003

Similar Records

A comprehensive and fair comparison of two neural operators (with practical extensions) based on $\mathrm{FAIR}$ data

Journal Article · Thu Mar 10 23:00:00 EST 2022 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:1976975

MIONet: Learning Multiple-Input Operators via Tensor Product

Journal Article · Sun Nov 06 23:00:00 EST 2022 · SIAM Journal on Scientific Computing · OSTI ID:2527397

Approximation rates of DeepONets for learning operators arising from advection–diffusion equations

Journal Article · Sat Jun 25 00:00:00 EDT 2022 · Neural Networks · OSTI ID:1977482

Related Subjects

97 MATHEMATICS AND COMPUTING
applied mathematics
computational science

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Citation Formats

References (29)

Similar Records

Related Subjects