Scalable Bayesian optimization with randomized prior networks

Bhouri, Mohamed Aziz; Joly, Michael; Yu, Robert; Sarkar, Soumalya; Perdikaris, Paris

doi:10.1016/j.cma.2023.116428

Title: Scalable Bayesian optimization with randomized prior networks

Journal Article · 22 September 2023 · Computer Methods in Applied Mechanics and Engineering

DOI: https://doi.org/10.1016/j.cma.2023.116428 · OSTI ID:2339523

^[1]; Joly, Michael ^[2]; Yu, Robert ^[3]; Sarkar, Soumalya ^[2]; Perdikaris, Paris ^[3]

Univ. of Pennsylvania, Philadelphia, PA (United States); University of Pennsylvania
RTX Technologies Research Center, Hartford, CT (United States)
Univ. of Pennsylvania, Philadelphia, PA (United States)

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Univ. of Pennsylvania, Philadelphia, PA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); US Air Force Office of Scientific Research (AFOSR); USDOE Advanced Research Projects Agency - Energy (ARPA-E); National Science Foundation (NSF)

Grant/Contract Number:: SC0019116; AR0001201

OSTI ID:: 2339523

Journal Information:: Computer Methods in Applied Mechanics and Engineering, Journal Name: Computer Methods in Applied Mechanics and Engineering Vol. 417; ISSN 0045-7825

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (33)

Induction of decision trees Quinlan, J. R. Machine Learning, Vol. 1, Issue 1, p. 81-106 https://doi.org/10.1007/BF00116251	journal	March 1986
Random variables with moment-matching staircase density functions Crespo, Luis G.; Kenny, Sean P.; Giesy, Daniel P. Applied Mathematical Modelling, Vol. 64 https://doi.org/10.1016/j.apm.2018.07.029	journal	December 2018
Calibration of a radiation quality model for sparse and uncertain data Crespo, Luis G.; Slaba, Tony C.; Kenny, Sean P. Applied Mathematical Modelling, Vol. 95 https://doi.org/10.1016/j.apm.2021.01.055	journal	July 2021
Multi-fidelity classification using Gaussian processes: Accelerating the prediction of large-scale computational models Sahli Costabal, Francisco; Perdikaris, Paris; Kuhl, Ellen Computer Methods in Applied Mechanics and Engineering, Vol. 357 https://doi.org/10.1016/j.cma.2019.112602	journal	December 2019
Bayesian optimization with output-weighted optimal sampling Blanchard, Antoine; Sapsis, Themistoklis Journal of Computational Physics, Vol. 425 https://doi.org/10.1016/j.jcp.2020.109901	journal	January 2021
Surrogate-based analysis and optimization Queipo, Nestor V.; Haftka, Raphael T.; Shyy, Wei Progress in Aerospace Sciences, Vol. 41, Issue 1 https://doi.org/10.1016/j.paerosci.2005.02.001	journal	January 2005
Recent advances in surrogate-based optimization Forrester, Alexander I. J.; Keane, Andy J. Progress in Aerospace Sciences, Vol. 45, Issue 1-3 https://doi.org/10.1016/j.paerosci.2008.11.001	journal	January 2009
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network Sherstinsky, Alex Physica D: Nonlinear Phenomena, Vol. 404 https://doi.org/10.1016/j.physd.2019.132306	journal	March 2020
On the quantification of aleatory and epistemic uncertainty using Sliced-Normal distributions Crespo, Luis G.; Colbert, Brendon K.; Kenny, Sean P. Systems & Control Letters, Vol. 134 https://doi.org/10.1016/j.sysconle.2019.104560	journal	December 2019
A perspective on regression and Bayesian approaches for system identification of pattern formation dynamics Wang, Zhenlin; Wu, Bowei; Garikipati, Krishna Theoretical and Applied Mechanics Letters, Vol. 10, Issue 3 https://doi.org/10.1016/j.taml.2020.01.028	journal	March 2020
Physics-constrained bayesian neural network for fluid flow reconstruction with sparse and noisy data Sun, Luning; Wang, Jian-Xun Theoretical and Applied Mechanics Letters, Vol. 10, Issue 3 https://doi.org/10.1016/j.taml.2020.01.031	journal	March 2020
Bayesian system identification of dynamical systems using highly informative training data Green, P. L.; Cross, E. J.; Worden, K. Mechanical Systems and Signal Processing, Vol. 56-57 https://doi.org/10.1016/j.ymssp.2014.10.003	journal	May 2015
On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression Nayek, R.; Fuentes, R.; Worden, K. Mechanical Systems and Signal Processing, Vol. 161 https://doi.org/10.1016/j.ymssp.2021.107986	journal	December 2021
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
Array programming with NumPy Harris, Charles R.; Millman, K. Jarrod; van der Walt, Stéfan J. Nature, Vol. 585, Issue 7825 https://doi.org/10.1038/s41586-020-2649-2	journal	September 2020
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators Lu, Lu; Jin, Pengzhan; Pang, Guofei Nature Machine Intelligence, Vol. 3, Issue 3 https://doi.org/10.1038/s42256-021-00302-5	journal	March 2021
Constrained Bayesian optimization for automatic chemical design using variational autoencoders Griffiths, Ryan-Rhys; Hernández-Lobato, José Miguel Chemical Science, Vol. 11, Issue 2 https://doi.org/10.1039/C9SC04026A	journal	January 2020
Sparsifying priors for Bayesian uncertainty quantification in model discovery Hirsh, Seth M.; Barajas-Solano, David A.; Kutz, J. Nathan Royal Society Open Science, Vol. 9, Issue 2 https://doi.org/10.1098/rsos.211823	journal	February 2022
Bayesian differential programming for robust systems identification under uncertainty Yang, Yibo; Aziz Bhouri, Mohamed; Perdikaris, Paris Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 476, Issue 2243 https://doi.org/10.1098/rspa.2020.0290	journal	November 2020
Gaussian processes meet NeuralODEs: a Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data Bhouri, Mohamed Aziz; Perdikaris, Paris Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 380, Issue 2229 https://doi.org/10.1098/rsta.2021.0201	journal	June 2022
Taking the Human Out of the Loop: A Review of Bayesian Optimization Shahriari, Bobak; Swersky, Kevin; Wang, Ziyu Proceedings of the IEEE, Vol. 104, Issue 1 https://doi.org/10.1109/JPROC.2015.2494218	journal	January 2016
An Efficient Batch-Constrained Bayesian Optimization Approach for Analog Circuit Synthesis via Multiobjective Acquisition Ensemble Zhang, Shuhan; Yang, Fan; Yan, Changhao IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 41, Issue 1 https://doi.org/10.1109/TCAD.2021.3054811	journal	January 2022
ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems Knowles, J. IEEE Transactions on Evolutionary Computation, Vol. 10, Issue 1 https://doi.org/10.1109/TEVC.2005.851274	journal	February 2006
Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels Emmerich, M. T. M.; Giannakoglou, K. C.; Naujoks, B. IEEE Transactions on Evolutionary Computation, Vol. 10, Issue 4 https://doi.org/10.1109/TEVC.2005.859463	journal	August 2006
A Review of Modern Computational Algorithms for Bayesian Optimal Design: A Review of Modern Algorithms for Bayesian Design Ryan, Elizabeth G.; Drovandi, Christopher C.; McGree, James M. International Statistical Review, Vol. 84, Issue 1 https://doi.org/10.1111/insr.12107	journal	June 2015
A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise Kushner, H. J. Journal of Basic Engineering, Vol. 86, Issue 1 https://doi.org/10.1115/1.3653121	journal	March 1964
Machine Learning Enabled Adaptive Optimization of a Transonic Compressor Rotor With Precompression Joly, Michael; Sarkar, Soumalya; Mehta, Dhagash Journal of Turbomachinery, Vol. 141, Issue 5 https://doi.org/10.1115/1.4041808	journal	January 2019
A Knowledge-Gradient Policy for Sequential Information Collection Frazier, Peter I.; Powell, Warren B.; Dayanik, Savas SIAM Journal on Control and Optimization, Vol. 47, Issue 5 https://doi.org/10.1137/070693424	journal	January 2008
Long Short-Term Memory Hochreiter, Sepp; Schmidhuber, Jürgen Neural Computation, Vol. 9, Issue 8 https://doi.org/10.1162/neco.1997.9.8.1735	journal	November 1997
Brief review of image denoising techniques Fan, Linwei; Zhang, Fan; Fan, Hui Visual Computing for Industry, Biomedicine, and Art, Vol. 2, Issue 1 https://doi.org/10.1186/s42492-019-0016-7	journal	July 2019
Bayesian Calibration and Uncertainty Analysis for Computationally Expensive Models Using Optimization and Radial Basis Function Approximation Bliznyuk, Nikolay; Ruppert, David; Shoemaker, Christine Journal of Computational and Graphical Statistics, Vol. 17, Issue 2 https://doi.org/10.1198/106186008X320681	journal	June 2008
py-pde: A Python package for solving partial differential equations Zwicker, David Journal of Open Source Software, Vol. 5, Issue 48 https://doi.org/10.21105/joss.02158	journal	April 2020
Wing Design by Numerical Optimization Hicks, Raymond M.; Henne, Preston A. Journal of Aircraft, Vol. 15, Issue 7 https://doi.org/10.2514/3.58379	journal	July 1978

Similar Records

srMO-BO-3GP: A sequential regularized multi-objective Bayesian optimization for constrained design applications using an uncertain Pareto classifier

Journal Article · 2021 · Journal of Mechanical Design · OSTI ID:1820417

Tran, Anh; Eldred, Michael; McCann, Scott; +1 more

A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Journal Article · 2021 · Journal of Mechanical Design · OSTI ID:1980664

Biswas, Arpan; Fuentes, Claudio; Hoyle, Christopher

A physics informed bayesian optimization approach for material design: application to NiTi shape memory alloys

Journal Article · 2023 · npj Computational Materials · OSTI ID:2578031

Khatamsaz, Danial; Neuberger, Raymond; Roy, Arunabha; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Monte Carlo approximation
constrained Bayesian optimization
deep ensembles
multi-fidelity Bayesian optimization
parallel acquisition functions
sequential decision making

Title: Scalable Bayesian optimization with randomized prior networks

Citation Formats

References (33)

Similar Records

Related Subjects