Multilevel Monte Carlo Estimators For Derivative-Free Optimization Under Uncertainty

Menhorn, Friedrich; Geraci, Gianluca; Seidl, Daniel Thomas; Marzouk, Youssef M.; Eldred, Michael S.; Bungartz, Hans-Joachim

doi:10.1615/int.j.uncertaintyquantification.2023048049

Multilevel Monte Carlo Estimators For Derivative-Free Optimization Under Uncertainty

Journal Article · Mon Jan 01 00:00:00 EST 2024 · International Journal for Uncertainty Quantification

DOI:https://doi.org/10.1615/int.j.uncertaintyquantification.2023048049· OSTI ID:2311583

Menhorn, Friedrich ^[1]; Geraci, Gianluca ^[2]; Seidl, Daniel Thomas ^[2]; Marzouk, Youssef M. ^[3]; Eldred, Michael S. ^[2]; Bungartz, Hans-Joachim ^[1]

Technical Univ. of Munich (Germany)
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)

Optimization is a key tool for scientific and engineering applications; however, in the presence of models affected by uncertainty, the optimization formulation needs to be extended to consider statistics of the quantity of interest. Optimization under uncertainty (OUU) deals with this endeavor and requires uncertainty quantification analyses at several design locations; i.e., its overall computational cost is proportional to the cost of performing a forward uncertainty analysis at each design location. An OUU workflow has two main components: an inner loop strategy for the computation of statistics of the quantity of interest, and an outer loop optimization strategy tasked with finding the optimal design, given a merit function based on the inner loop statistics. Here, in this work, we propose to alleviate the cost of the inner loop uncertainty analysis by leveraging the so-called multilevel Monte Carlo (MLMC) method, which is able to allocate resources over multiple models with varying accuracy and cost. The resource allocation problem in MLMC is formulated by minimizing the computational cost given a target variance for the estimator. We consider MLMC estimators for statistics usually employed in OUU workflows and solve the corresponding allocation problem. For the outer loop, we consider a derivative-free optimization strategy implemented in the SNOWPAC library; our novel strategy is implemented and released in the Dakota software toolkit. We discuss several numerical test cases to showcase the features and performance of our approach with respect to its Monte Carlo single fidelity counterpart.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR). Scientific Discovery through Advanced Computing (SciDAC)

Grant/Contract Number:: NA0003525

OSTI ID:: 2311583

Report Number(s):: SAND--2024-00215J

Journal Information:: International Journal for Uncertainty Quantification, Journal Name: International Journal for Uncertainty Quantification Journal Issue: 3 Vol. 14; ISSN 2152-5080

Publisher:: Begell HouseCopyright Statement

Country of Publication:: United States

Language:: English

References (74)

Appendix A: The Delta Method Liu, Xian Survival Analysis https://doi.org/10.1002/9781118307656.app1	journal	July 2012
Using quaternions to calculate RMSD Coutsias, Evangelos A.; Seok, Chaok; Dill, Ken A. Journal of Computational Chemistry, Vol. 25, Issue 15 https://doi.org/10.1002/jcc.20110	journal	January 2004
Multifidelity approaches for optimization under uncertainty: MULTIFIDELITY APPROACHES FOR OPTIMIZATION UNDER UNCERTAINTY Ng, Leo W. T.; Willcox, Karen E. International Journal for Numerical Methods in Engineering, Vol. 100, Issue 10 https://doi.org/10.1002/nme.4761	journal	September 2014
Multilevel Monte Carlo Methods Heinrich, Stefan Large-Scale Scientific Computing https://doi.org/10.1007/3-540-45346-6_5	book	January 2001
The Simplex Gradient and Noisy Optimization Problems Bortz, D. M.; Kelley, C. T. Computational Methods for Optimal Design and Control https://doi.org/10.1007/978-1-4612-1780-0_5	book	January 1998
Optimization of Stochastic Models Pflug, George Ch. The Kluwer International Series in Engineering and Computer Science https://doi.org/10.1007/978-1-4613-1449-3	book	January 1996
Introduction to the Theory of Probabilistic Functions and Percentiles (Value-at-Risk) Uryasev, S. Nonconvex Optimization and Its Applications https://doi.org/10.1007/978-1-4757-3150-7_1	book	January 2000
Optimization Under Uncertainty Diwekar, Urmila M. Applied Optimization https://doi.org/10.1007/978-1-4757-3745-5_5	book	January 2003
Constrained multifidelity optimization using model calibration March, Andrew; Willcox, Karen Structural and Multidisciplinary Optimization, Vol. 46, Issue 1 https://doi.org/10.1007/s00158-011-0749-1	journal	January 2012
Multi-index Monte Carlo: when sparsity meets sampling Haji-Ali, Abdul-Lateef; Nobile, Fabio; Tempone, Raúl Numerische Mathematik, Vol. 132, Issue 4 https://doi.org/10.1007/s00211-015-0734-5	journal	June 2015
MFNets: data efficient all-at-once learning of multifidelity surrogates as directed networks of information sources Gorodetsky, A. A.; Jakeman, J. D.; Geraci, G. Computational Mechanics, Vol. 68, Issue 4 https://doi.org/10.1007/s00466-021-02042-0	journal	August 2021
Stochastic optimization using a trust-region method and random models Chen, R.; Menickelly, M.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1141-8	journal	April 2017
A derivative-free trust-funnel method for equality-constrained nonlinear optimization Sampaio, Ph. R.; Toint, Ph. L. Computational Optimization and Applications, Vol. 61, Issue 1 https://doi.org/10.1007/s10589-014-9715-3	journal	November 2014
Stochastic derivative-free optimization using a trust region framework Larson, Jeffrey; Billups, Stephen C. Computational Optimization and Applications, Vol. 64, Issue 3 https://doi.org/10.1007/s10589-016-9827-z	journal	February 2016
General Robust-Optimization Formulation for Nonlinear Programming Zhang, Y. Journal of Optimization Theory and Applications, Vol. 132, Issue 1 https://doi.org/10.1007/s10957-006-9082-z	journal	February 2007
A Multi Level Monte Carlo method with control variate for elliptic PDEs with log-normal coefficients Nobile, Fabio; Tesei, Francesco Stochastic Partial Differential Equations: Analysis and Computations, Vol. 3, Issue 3 https://doi.org/10.1007/s40072-015-0055-9	journal	July 2015
Estimation of arbitrary order central statistical moments by the multilevel Monte Carlo method Bierig, Claudio; Chernov, Alexey Stochastics and Partial Differential Equations Analysis and Computations, Vol. 4, Issue 1 https://doi.org/10.1007/s40072-015-0063-9	journal	October 2015
Robust design optimization of structures through consideration of variation Sandgren, Eric; Cameron, T. M. Computers & Structures, Vol. 80, Issue 20-21 https://doi.org/10.1016/S0045-7949(02)00160-8	journal	August 2002
Reliability analysis—a review and some perspectives Rackwitz, Rüdiger Structural Safety, Vol. 23, Issue 4 https://doi.org/10.1016/S0167-4730(02)00009-7	journal	October 2001
Robust solutions of uncertain linear programs Ben-Tal, A.; Nemirovski, A. Operations Research Letters, Vol. 25, Issue 1 https://doi.org/10.1016/S0167-6377(99)00016-4	journal	August 1999
Measures of risk Szegö, Giorgio Journal of Banking & Finance, Vol. 26, Issue 7 https://doi.org/10.1016/S0378-4266(02)00262-5	journal	July 2002
Conditional value-at-risk for general loss distributions Rockafellar, R. Tyrrell; Uryasev, Stanislav Journal of Banking & Finance, Vol. 26, Issue 7 https://doi.org/10.1016/S0378-4266(02)00271-6	journal	July 2002
Robust optimization – A comprehensive survey Beyer, Hans-Georg; Sendhoff, Bernhard Computer Methods in Applied Mechanics and Engineering, Vol. 196, Issue 33-34 https://doi.org/10.1016/j.cma.2007.03.003	journal	July 2007
Multifidelity uncertainty quantification with models based on dissimilar parameters Zeng, Xiaoshu; Geraci, Gianluca; Eldred, Michael S. Computer Methods in Applied Mechanics and Engineering, Vol. 415 https://doi.org/10.1016/j.cma.2023.116205	journal	October 2023
A review of robust optimal design and its application in dynamics Zang, C.; Friswell, M. I.; Mottershead, J. E. Computers & Structures, Vol. 83, Issue 4-5 https://doi.org/10.1016/j.compstruc.2004.10.007	journal	January 2005
Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions Regis, Rommel G. Computers & Operations Research, Vol. 38, Issue 5 https://doi.org/10.1016/j.cor.2010.09.013	journal	May 2011
A low-rank control variate for multilevel Monte Carlo simulation of high-dimensional uncertain systems Fairbanks, Hillary R.; Doostan, Alireza; Ketelsen, Christian Journal of Computational Physics, Vol. 341 https://doi.org/10.1016/j.jcp.2017.03.060	journal	July 2017
A generalized approximate control variate framework for multifidelity uncertainty quantification Gorodetsky, Alex A.; Geraci, Gianluca; Eldred, Michael S. Journal of Computational Physics, Vol. 408 https://doi.org/10.1016/j.jcp.2020.109257	journal	May 2020
Quantifying uncertain system outputs via the multilevel Monte Carlo method — Part I: Central moment estimation Krumscheid, S.; Nobile, F.; Pisaroni, M. Journal of Computational Physics, Vol. 414 https://doi.org/10.1016/j.jcp.2020.109466	journal	August 2020
Gradient-based optimisation of the conditional-value-at-risk using the multi-level Monte Carlo method Ganesh, Sundar; Nobile, Fabio Journal of Computational Physics, Vol. 495 https://doi.org/10.1016/j.jcp.2023.112523	journal	December 2023
Review of uncertainty-based multidisciplinary design optimization methods for aerospace vehicles Yao, Wen; Chen, Xiaoqian; Luo, Wencai Progress in Aerospace Sciences, Vol. 47, Issue 6 https://doi.org/10.1016/j.paerosci.2011.05.001	journal	August 2011
Modeling and optimization of risk Krokhmal, Pavlo; Zabarankin, Michael; Uryasev, Stan Surveys in Operations Research and Management Science, Vol. 16, Issue 2 https://doi.org/10.1016/j.sorms.2010.08.001	journal	July 2011
Computer Age Statistical Inference Efron, Bradley; Hastie, Trevor https://doi.org/10.1017/CBO9781316576533	book	July 2016
Multilevel Monte Carlo methods Giles, Michael B. Acta Numerica, Vol. 24 https://doi.org/10.1017/S096249291500001X	journal	April 2015
Enhanced solvation force extrapolation for speeding up molecular dynamics simulations of complex biochemical liquids Omelyan, Igor; Kovalenko, Andriy The Journal of Chemical Physics, Vol. 151, Issue 21 https://doi.org/10.1063/1.5126410	journal	December 2019
The Complications of the Fourth Central Moment Dodge, Yadolah; Rousson, Valentin The American Statistician, Vol. 53, Issue 3 https://doi.org/10.1080/00031305.1999.10474471	journal	August 1999
Some Useful Moment Results in Sampling Problems O’Neill, B. The American Statistician, Vol. 68, Issue 4 https://doi.org/10.1080/00031305.2014.966589	journal	October 2014
Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points Regis, Rommel G. Engineering Optimization, Vol. 46, Issue 2 https://doi.org/10.1080/0305215X.2013.765000	journal	February 2013
Control-variate estimation using estimated control means Pasupathy, Raghu; Schmeiser, Bruce W.; Taaffe, Michael R. IIE Transactions, Vol. 44, Issue 5 https://doi.org/10.1080/0740817X.2011.610430	journal	May 2012
Predicting the output from a complex computer code when fast approximations are available Kennedy, M. Biometrika, Vol. 87, Issue 1 https://doi.org/10.1093/biomet/87.1.1	journal	March 2000
Coherent Measures of Risk Artzner, Philippe; Delbaen, Freddy; Eber, Jean-Marc Mathematical Finance, Vol. 9, Issue 3 https://doi.org/10.1111/1467-9965.00068	journal	July 1999
Expected Shortfall: A Natural Coherent Alternative to Value at Risk Acerbi, Carlo; Tasche, Dirk Economic Notes, Vol. 31, Issue 2 https://doi.org/10.1111/1468-0300.00091	journal	July 2002
New Concepts in Graphic Visualization of Objective Functions Simionescu, Petru-Aurelian; Beale, David G. Volume 2: 28th Design Automation Conference https://doi.org/10.1115/DETC2002/DAC-34129	conference	January 2002
Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points Conn, Andrew R.; Scheinberg, Katya; Vicente, Luís N. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/060673424	journal	January 2009
On the Global Convergence of Trust Region Algorithms Using Inexact Gradient Information Carter, Richard G. SIAM Journal on Numerical Analysis, Vol. 28, Issue 1 https://doi.org/10.1137/0728014	journal	February 1991
Global Convergence of a Class of Trust Region Algorithms for Optimization Using Inexact Projections on Convex Constraints Conn, A. R.; Gould, Nick; Sartenaer, A. SIAM Journal on Optimization, Vol. 3, Issue 1 https://doi.org/10.1137/0803009	journal	February 1993
Theory and Applications of Robust Optimization Bertsimas, Dimitris; Brown, David B.; Caramanis, Constantine SIAM Review, Vol. 53, Issue 3 https://doi.org/10.1137/080734510	journal	January 2011
Trust Region Methods Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe L. MOS-SIAM Series on Optimization https://doi.org/10.1137/1.9780898719857	book	January 2000
Iterative Methods for Optimization Kelley, C. T. https://doi.org/10.1137/1.9781611970920	book	January 1999
Optimal Model Management for Multifidelity Monte Carlo Estimation Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max SIAM Journal on Scientific Computing, Vol. 38, Issue 5 https://doi.org/10.1137/15M1046472	journal	January 2016
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
Survey of Multifidelity Methods in Uncertainty Propagation, Inference, and Optimization Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max SIAM Review, Vol. 60, Issue 3 https://doi.org/10.1137/16M1082469	journal	January 2018
Multifidelity Monte Carlo Estimation of Variance and Sensitivity Indices Qian, E.; Peherstorfer, B.; O'Malley, D. SIAM/ASA Journal on Uncertainty Quantification, Vol. 6, Issue 2 https://doi.org/10.1137/17M1151006	journal	January 2018
On Multilevel Best Linear Unbiased Estimators Schaden, Daniel; Ullmann, Elisabeth SIAM/ASA Journal on Uncertainty Quantification, Vol. 8, Issue 2 https://doi.org/10.1137/19M1263534	journal	January 2020
Asymptotic Analysis of Multilevel Best Linear Unbiased Estimators Schaden, Daniel; Ullmann, Elisabeth SIAM/ASA Journal on Uncertainty Quantification, Vol. 9, Issue 3 https://doi.org/10.1137/20M1321607	journal	January 2021
Superlinear Convergence and Implicit Filtering Choi, T. D.; Kelley, C. T. SIAM Journal on Optimization, Vol. 10, Issue 4 https://doi.org/10.1137/S1052623499354096	journal	January 2000
Analysis of Inexact Trust-Region SQP Algorithms Heinkenschloss, Matthias; Vicente, Luis N. SIAM Journal on Optimization, Vol. 12, Issue 2 https://doi.org/10.1137/S1052623499361543	journal	January 2002
Sample Mean and Sample Variance Zhang, Lingyun The American Statistician, Vol. 61, Issue 2 https://doi.org/10.1198/000313007X188379	journal	May 2007
An Introduction to the Bootstrap Efron, Bradley; Tibshirani, R. J. Monographs on Statistics and Applied Probability https://doi.org/10.1201/9780429246593	book	May 1994
Stochastic Estimation of the Maximum of a Regression Function Kiefer, J.; Wolfowitz, J. The Annals of Mathematical Statistics, Vol. 23, Issue 3 https://doi.org/10.1214/aoms/1177729392	journal	September 1952
A Stochastic Approximation Method Robbins, Herbert; Monro, Sutton The Annals of Mathematical Statistics, Vol. 22, Issue 3 https://doi.org/10.1214/aoms/1177729586	journal	September 1951
Multilevel Monte Carlo Path Simulation Giles, Michael B. Operations Research, Vol. 56, Issue 3 https://doi.org/10.1287/opre.1070.0496	journal	June 2008
Recursive Co-Kriging Model for Design of Computer Experiments with Multiple Levels of Fidelity Le Gratiet, Loic; Garnier, Josselin International Journal for Uncertainty Quantification, Vol. 4, Issue 5 https://doi.org/10.1615/Int.J.UncertaintyQuantification.2014006914	journal	January 2014
MFNets: MULTI-FIDELITY DATA-DRIVEN NETWORKS FOR BAYESIAN LEARNING AND PREDICTION Gorodetsky, Alex A.; Jakeman, John D.; Geraci, Gianluca International Journal for Uncertainty Quantification, Vol. 10, Issue 6 https://doi.org/10.1615/Int.J.UncertaintyQuantification.2020032978	journal	January 2020
Optimization of conditional value-at-risk Rockafellar, R. Tyrrell; Uryasev, Stanislav The Journal of Risk, Vol. 2, Issue 3 https://doi.org/10.21314/JOR.2000.038	journal	January 2000
Deviation Measures in Risk Analysis and Optimization Rockafellar, Tyrrell R.; Uryasev, Stanislav P.; Zabarankin, Michael SSRN Electronic Journal https://doi.org/10.2139/ssrn.365640	journal	January 2003
Adaptive multi-index collocation for uncertainty quantification and sensitivity analysis Jakeman, John Davis; Eldred, Michael S.; Geraci, G. https://doi.org/10.2172/1574406	report	November 2019
Dakota, A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis (V.6.16 User's Manual) Adams, Brian M.; Bohnhoff, William J.; Dalbey, Keith R. https://doi.org/10.2172/1868142	report	November 2021
Dakota, A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.16 Theory Manual Dalbey, Keith R.; Eldred, Michael S.; Geraci, Gianluca https://doi.org/10.2172/1868423	report	January 2021
Robust and Reliability-Based Design Optimization Framework for Wing Design Paiva, Ricardo M.; Crawford, Curran; Suleman, Afzal AIAA Journal, Vol. 52, Issue 4 https://doi.org/10.2514/1.J052161	journal	April 2014
Multifidelity Sparse Polynomial Chaos Surrogate Models Applied to Flutter Databases Rumpfkeil, Markus P.; Beran, Philip S. AIAA Journal, Vol. 58, Issue 3 https://doi.org/10.2514/1.J058452	journal	March 2020
Multimodal Reliability Assessment for Complex Engineering Applications using Efficient Global Optimization Bichon, Barron; Eldred, Michael; Swiler, Laura 48th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference https://doi.org/10.2514/6.2007-1946	conference	April 2007
A multifidelity multilevel Monte Carlo method for uncertainty propagation in aerospace applications Geraci, Gianluca; Eldred, Michael S.; Iaccarino, Gianluca 19th AIAA Non-Deterministic Approaches Conference https://doi.org/10.2514/6.2017-1951	conference	January 2017
Gaussian Processes for Machine Learning Rasmussen, Carl Edward; Williams, Christopher K. I. The MIT Press https://doi.org/10.7551/mitpress/3206.001.0001	book	January 2005

Similar Records

Estimation of distributions via multilevel Monte Carlo with stratified sampling

Journal Article · Wed Jul 08 20:00:00 EDT 2020 · Journal of Computational Physics · OSTI ID:1852855

An improved multilevel Monte Carlo method for estimating probability distribution functions in stochastic oil reservoir simulations

Journal Article · Thu Dec 29 19:00:00 EST 2016 · Water Resources Research · OSTI ID:1338482

Related Subjects

97 MATHEMATICS AND COMPUTING
multilevel Monte Carlo
optimization under uncertainty
uncertainty quantification

Multilevel Monte Carlo Estimators For Derivative-Free Optimization Under Uncertainty

Citation Formats

References (74)

Similar Records

Related Subjects