Probabilistic partition of unity networks for&nbsp;high–dimensional regression problems

Fan, Tiffany; Trask, Nathaniel; D'Elia, Marta; Darve, Eric

doi:10.1002/nme.7207

Probabilistic partition of unity networks for high–dimensional regression problems

Journal Article · Thu Jan 26 23:00:00 EST 2023 · International Journal for Numerical Methods in Engineering

DOI:https://doi.org/10.1002/nme.7207· OSTI ID:2305532

^[1]; Trask, Nathaniel ^[2]; D'Elia, Marta ^[3]; ^[4]

Stanford University, CA (United States); Stanford University
Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
Sandia National Laboratories (SNL-CA), Livermore, CA (United States)
Stanford University, CA (United States)

We explore the probabilistic partition of unity network (PPOU-Net) model in the context of high-dimensional regression problems and propose a general framework focusing on adaptive dimensionality reduction. With the proposed framework, the target function is approximated by a mixture of experts model on a low-dimensional manifold, where each cluster is associated with a fixed-degree polynomial. We present a training strategy that leverages the expectation maximization (EM) algorithm. During the training, we alternate between (i) applying gradient descent to update the DNN coefficients; and (ii) using closed-form formulae derived from the EM algorithm to update the mixture of experts model parameters. Under the probabilistic formulation, step (ii) admits the form of embarrassingly paralleliazable weighted least-squares solves. The PPOU-Nets consistently outperform the baseline fully-connected neural networks of comparable sizes in numerical experiments of various data dimensions. Here, we also explore the proposed model in applications of quantum computing, where the PPOU-Nets act as surrogate models for cost landscapes associated with variational quantum circuits.

View Accepted Manuscript (DOE)

Research Organization:: Stanford University, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0019205; NA0003525

OSTI ID:: 2305532

Journal Information:: International Journal for Numerical Methods in Engineering, Journal Name: International Journal for Numerical Methods in Engineering Journal Issue: 10 Vol. 124; ISSN 0029-5981

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (48)

Mixture of experts: a literature survey Masoudnia, Saeed; Ebrahimpour, Reza Artificial Intelligence Review, Vol. 42, Issue 2 https://doi.org/10.1007/s10462-012-9338-y	journal	May 2012
Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review Poggio, Tomaso; Mhaskar, Hrushikesh; Rosasco, Lorenzo International Journal of Automation and Computing, Vol. 14, Issue 5 https://doi.org/10.1007/s11633-017-1054-2	journal	March 2017
Learning generative neural networks with physics knowledge Xu, Kailai; Zhu, Weiqiang; Darve, Eric Research in the Mathematical Sciences, Vol. 9, Issue 2 https://doi.org/10.1007/s40687-022-00329-z	journal	May 2022
Convergence results for the EM approach to mixtures of experts architectures Jordan, Michael I.; Xu, Lei Neural Networks, Vol. 8, Issue 9 https://doi.org/10.1016/0893-6080(95)00014-3	journal	January 1995
Filling the gaps: Gaussian mixture models from noisy, truncated or incomplete samples Melchior, P.; Goulding, A. D. Astronomy and Computing, Vol. 25 https://doi.org/10.1016/j.ascom.2018.09.013	journal	October 2018
Non-intrusive low-rank separated approximation of high-dimensional stochastic models Doostan, Alireza; Validi, AbdoulAhad; Iaccarino, Gianluca Computer Methods in Applied Mechanics and Engineering, Vol. 263 https://doi.org/10.1016/j.cma.2013.04.003	journal	August 2013
Data-driven computational mechanics Kirchdoerfer, T.; Ortiz, M. Computer Methods in Applied Mechanics and Engineering, Vol. 304 https://doi.org/10.1016/j.cma.2016.02.001	journal	June 2016
Exploiting active subspaces to quantify uncertainty in the numerical simulation of the HyShot II scramjet Constantine, P. G.; Emory, M.; Larsson, J. Journal of Computational Physics, Vol. 302 https://doi.org/10.1016/j.jcp.2015.09.001	journal	December 2015
Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification Zhu, Yinhao; Zabaras, Nicholas Journal of Computational Physics, Vol. 366 https://doi.org/10.1016/j.jcp.2018.04.018	journal	August 2018
Gradient-based optimization for regression in the functional tensor-train format Gorodetsky, Alex A.; Jakeman, John D. Journal of Computational Physics, Vol. 374 https://doi.org/10.1016/j.jcp.2018.08.010	journal	December 2018
DGM: A deep learning algorithm for solving partial differential equations Sirignano, Justin; Spiliopoulos, Konstantinos Journal of Computational Physics, Vol. 375 https://doi.org/10.1016/j.jcp.2018.08.029	journal	December 2018
Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification Tripathy, Rohit K.; Bilionis, Ilias Journal of Computational Physics, Vol. 375 https://doi.org/10.1016/j.jcp.2018.08.036	journal	December 2018
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.024	journal	October 2019
Learning constitutive relations from indirect observations using deep neural networks Huang, Daniel Z.; Xu, Kailai; Farhat, Charbel Journal of Computational Physics, Vol. 416 https://doi.org/10.1016/j.jcp.2020.109491	journal	September 2020
Physics constrained learning for data-driven inverse modeling from sparse observations Xu, Kailai; Darve, Eric Journal of Computational Physics, Vol. 453 https://doi.org/10.1016/j.jcp.2021.110938	journal	March 2022
A unified deep artificial neural network approach to partial differential equations in complex geometries Berg, Jens; Nyström, Kaj Neurocomputing, Vol. 317 https://doi.org/10.1016/j.neucom.2018.06.056	journal	November 2018
Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition Stallkamp, J.; Schlipsing, M.; Salmen, J. Neural Networks, Vol. 32 https://doi.org/10.1016/j.neunet.2012.02.016	journal	August 2012
Error bounds for approximations with deep ReLU networks Yarotsky, Dmitry Neural Networks, Vol. 94 https://doi.org/10.1016/j.neunet.2017.07.002	journal	October 2017
Kernel PCA for novelty detection Hoffmann, Heiko Pattern Recognition, Vol. 40, Issue 3 https://doi.org/10.1016/j.patcog.2006.07.009	journal	March 2007
Reliability analysis of high-dimensional models using low-rank tensor approximations Konakli, Katerina; Sudret, Bruno Probabilistic Engineering Mechanics, Vol. 46 https://doi.org/10.1016/j.probengmech.2016.08.002	journal	October 2016
Deep learning for high-dimensional reliability analysis Li, Mingyang; Wang, Zequn Mechanical Systems and Signal Processing, Vol. 139 https://doi.org/10.1016/j.ymssp.2019.106399	journal	May 2020
Radial Basis Functions Buhmann, Martin D. Cambridge University Press https://doi.org/10.1017/CBO9780511543241	book	January 2009
Solving parametric PDE problems with artificial neural networks Khoo, Yuehaw; Lu, Jianfeng; Ying, Lexing European Journal of Applied Mathematics, Vol. 32, Issue 3 https://doi.org/10.1017/S0956792520000182	journal	July 2020
Physics-informed machine learning Karniadakis, George Em; Kevrekidis, Ioannis G.; Lu, Lu Nature Reviews Physics, Vol. 3, Issue 6 https://doi.org/10.1038/s42254-021-00314-5	journal	May 2021
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators Lu, Lu; Jin, Pengzhan; Pang, Guofei Nature Machine Intelligence, Vol. 3, Issue 3 https://doi.org/10.1038/s42256-021-00302-5	journal	March 2021
Evaluation of machine learning algorithms for prediction of regions of high Reynolds averaged Navier Stokes uncertainty Ling, J.; Templeton, J. Physics of Fluids, Vol. 27, Issue 8 https://doi.org/10.1063/1.4927765	journal	August 2015
Solving high-dimensional partial differential equations using deep learning Han, Jiequn; Jentzen, Arnulf; E., Weinan Proceedings of the National Academy of Sciences, Vol. 115, Issue 34 https://doi.org/10.1073/pnas.1718942115	journal	August 2018
Machine learning classification can reduce false positives in structure-based virtual screening Adeshina, Yusuf O.; Deeds, Eric J.; Karanicolas, John Proceedings of the National Academy of Sciences, Vol. 117, Issue 31 https://doi.org/10.1073/pnas.2000585117	journal	July 2020
Splines in Statistics Wegman, Edward J.; Wright, Ian W. Journal of the American Statistical Association, Vol. 78, Issue 382 https://doi.org/10.1080/01621459.1983.10477977	journal	June 1983
Variational Inference: A Review for Statisticians Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D. Journal of the American Statistical Association, Vol. 112, Issue 518 https://doi.org/10.1080/01621459.2017.1285773	journal	July 2016
Artificial neural networks for solving ordinary and partial differential equations Lagaris, I. E.; Likas, A.; Fotiadis, D. I. IEEE Transactions on Neural Networks, Vol. 9, Issue 5 https://doi.org/10.1109/72.712178	journal	January 1998
Distributed Signal Processing via Chebyshev Polynomial Approximation Shuman, David I.; Vandergheynst, Pierre; Kressner, Daniel IEEE Transactions on Signal and Information Processing over Networks, Vol. 4, Issue 4 https://doi.org/10.1109/TSIPN.2018.2824239	journal	December 2018
Reducing the Dimensionality of Data with Neural Networks Hinton, G. E. Science, Vol. 313, Issue 5786 https://doi.org/10.1126/science.1127647	journal	July 2006
A Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data Nobile, F.; Tempone, R.; Webster, C. G. SIAM Journal on Numerical Analysis, Vol. 46, Issue 5 https://doi.org/10.1137/060663660	journal	January 2008
Active Subspaces Constantine, Paul G. Society for Industrial and Applied Mathematics https://doi.org/10.1137/1.9781611973860	book	January 2015
Approximation Theory and Approximation Practice, Extended Edition Trefethen, Lloyd N. https://doi.org/10.1137/1.9781611975949	book	January 2019
Spectral Tensor-Train Decomposition Bigoni, Daniele; Engsig-Karup, Allan P.; Marzouk, Youssef M. SIAM Journal on Scientific Computing, Vol. 38, Issue 4 https://doi.org/10.1137/15M1036919	journal	January 2016
Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ Schwab, Christoph; Zech, Jakob Analysis and Applications, Vol. 17, Issue 01 https://doi.org/10.1142/S0219530518500203	journal	December 2018
Deep ReLU networks and high-order finite element methods Opschoor, Joost A. A.; Petersen, Philipp C.; Schwab, Christoph Analysis and Applications, Vol. 18, Issue 05 https://doi.org/10.1142/S0219530519410136	journal	February 2020
Adaptive Mixtures of Local Experts Jacobs, Robert A.; Jordan, Michael I.; Nowlan, Steven J. Neural Computation, Vol. 3, Issue 1 https://doi.org/10.1162/neco.1991.3.1.79	journal	February 1991
Hierarchical Mixtures of Experts and the EM Algorithm Jordan, Michael I.; Jacobs, Robert A. Neural Computation, Vol. 6, Issue 2 https://doi.org/10.1162/neco.1994.6.2.181	journal	March 1994
Lebesgue functions and Lebesgue constants in polynomial interpolation Ibrahimoglu, Bayram Ali Journal of Inequalities and Applications, Vol. 2016, Issue 1 https://doi.org/10.1186/s13660-016-1030-3	journal	March 2016
Projection-Based Approximation and a Duality with Kernel Methods Donoho, David L.; Johnstone, Iain M. The Annals of Statistics, Vol. 17, Issue 1 https://doi.org/10.1214/aos/1176347004	journal	March 1989
An Introduction to Variational Autoencoders Kingma, Diederik P.; Welling, Max Foundations and Trends® in Machine Learning, Vol. 12, Issue 4 https://doi.org/10.1561/2200000056	journal	January 2019
Extending Classical Surrogate Modeling to high Dimensions Through Supervised Dimensionality Reduction: a Data-Driven Approach Lataniotis, Christos; Marelli, Stefano; Sudret, Bruno International Journal for Uncertainty Quantification, Vol. 10, Issue 1 https://doi.org/10.1615/Int.J.UncertaintyQuantification.2020031935	journal	January 2020
Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence Baker, Nathan; Alexander, Frank; Bremer, Timo https://doi.org/10.2172/1478744	report	February 2019
Active Manifold and Model-Order Reduction to Accelerate Multidisciplinary Analysis and Optimization Boncoraglio, Gabriele; Farhat, Charbel AIAA Journal, Vol. 59, Issue 11 https://doi.org/10.2514/1.J060581	journal	November 2021

Similar Records

PhILMs: Collaboratory on Mathematics and Physics-Informed Learning Machines for Multiscale and Multiphysics Problems

Technical Report · Thu Feb 15 23:00:00 EST 2024 · OSTI ID:2305747

Related Subjects

97 MATHEMATICS AND COMPUTING
deep learning
dimensionality reduction
mixture of experts
nonparametric regression

Probabilistic partition of unity networks for high–dimensional regression problems

Citation Formats

References (48)

Similar Records

Related Subjects