Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation

Lyu, Xiong; Binois, Mickaël; Ludkovski, Michael

doi:10.1007/s11222-021-10014-w

Title: Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation

Journal Article · 26 May 2021 · Statistics and Computing

DOI: https://doi.org/10.1007/s11222-021-10014-w · OSTI ID:1784775

; Binois, Mickaël; Ludkovski, Michael

Abstract We consider the problem of learning the level set for which a noisy black-box function exceeds a given threshold. To efficiently reconstruct the level set, we investigate Gaussian process (GP) metamodels. Our focus is on strongly stochastic simulators, in particular with heavy-tailed simulation noise and low signal-to-noise ratio. To guard against noise misspecification, we assess the performance of three variants: (i) GPs with Student- t observations; (ii) Student- t processes (TPs); and (iii) classification GPs modeling the sign of the response. In conjunction with these metamodels, we analyze several acquisition functions for guiding the sequential experimental designs, extending existing stepwise uncertainty reduction criteria to the stochastic contour-finding context. This also motivates our development of (approximate) updating formulas to efficiently compute such acquisition functions. Our schemes are benchmarked by using a variety of synthetic experiments in 1–6 dimensions. We also consider an application of level set estimation for determining the optimal exercise policy of Bermudan options in finance.

View Journal Article

Cite

Export

Save

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1784775

Journal Information:: Statistics and Computing, Journal Name: Statistics and Computing Journal Issue: 4 Vol. 31; ISSN 0960-3174

Publisher:: Springer Science + Business MediaCopyright Statement

Country of Publication:: Germany

Language:: English

References (30)

A benchmark of kriging-based infill criteria for noisy optimization Picheny, Victor; Wagner, Tobias; Ginsbourger, David Structural and Multidisciplinary Optimization, Vol. 48, Issue 3 https://doi.org/10.1007/s00158-013-0919-4	journal	April 2013
Sequential design of computer experiments for the estimation of a probability of failure Bect, Julien; Ginsbourger, David; Li, Ling Statistics and Computing, Vol. 22, Issue 3 https://doi.org/10.1007/s11222-011-9241-4	journal	April 2011
Comparison of Kriging-based algorithms for simulation optimization with heterogeneous noise Jalali, Hamed; Van Nieuwenhuyse, Inneke; Picheny, Victor European Journal of Operational Research, Vol. 261, Issue 1 https://doi.org/10.1016/j.ejor.2017.01.035	journal	August 2017
Extended t -process regression models Wang, Zhanfeng; Shi, Jian Qing; Lee, Youngjo Journal of Statistical Planning and Inference, Vol. 189 https://doi.org/10.1016/j.jspi.2017.05.006	journal	October 2017
Efficient Global Optimization of Expensive Black-Box Functions Jones, Donald R.; Schonlau, Matthias; Welch, William J. Journal of Global Optimization, Vol. 13, Issue 4, p. 455-492 https://doi.org/10.1023/A:1008306431147	journal	January 1998
Quantile-Based Optimization of Noisy Computer Experiments With Tunable Precision Picheny, Victor; Ginsbourger, David; Richet, Yann Technometrics, Vol. 55, Issue 1 https://doi.org/10.1080/00401706.2012.707580	journal	July 2012
Fast Parallel Kriging-Based Stepwise Uncertainty Reduction With Application to the Identification of an Excursion Set Chevalier, Clément; Bect, Julien; Ginsbourger, David Technometrics, Vol. 56, Issue 4 https://doi.org/10.1080/00401706.2013.860918	journal	October 2014
Replication or Exploration? Sequential Design for Stochastic Simulation Experiments Binois, Mickaël; Huang, Jiangeng; Gramacy, Robert B. Technometrics, Vol. 61, Issue 1 https://doi.org/10.1080/00401706.2018.1469433	journal	September 2018
Practical Heteroscedastic Gaussian Process Modeling for Large Simulation Experiments Binois, Mickaël; Gramacy, Robert B.; Ludkovski, Mike Journal of Computational and Graphical Statistics, Vol. 27, Issue 4 https://doi.org/10.1080/10618600.2018.1458625	journal	July 2018
Valuing American Options by Simulation: A Simple Least-Squares Approach Longstaff, Francis A.; Schwartz, Eduardo S. Review of Financial Studies, Vol. 14, Issue 1 https://doi.org/10.1093/rfs/14.1.113	journal	January 2001
Bayesian classification with Gaussian processes Williams, C. K. I.; Barber, D. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, Issue 12 https://doi.org/10.1109/34.735807	journal	January 1998
Regression methods for pricing complex American-style options Tsitsiklis, J. N.; Van Roy, B. IEEE Transactions on Neural Networks, Vol. 12, Issue 4 https://doi.org/10.1109/72.935083	journal	July 2001
Level set estimation with dynamic sparse sensing Yang, Jing; Wang, Zuoen; Wu, Jingxian 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP) https://doi.org/10.1109/GlobalSIP.2014.7032165	conference	December 2014
Modeling of failure probability and statistical design of SRAM array for yield enhancement in nanoscaled CMOS Mukhopadhyay, S.; Mahmoodi, H.; Roy, K. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 24, Issue 12 https://doi.org/10.1109/TCAD.2005.852295	journal	December 2005
Minimax Optimal Level-Set Estimation Willett, R. M.; Nowak, R. D. IEEE Transactions on Image Processing, Vol. 16, Issue 12 https://doi.org/10.1109/TIP.2007.910175	journal	December 2007
Information-Theoretic Regret Bounds for Gaussian Process Optimization in the Bandit Setting Srinivas, Niranjan; Krause, Andreas; Kakade, Sham M. IEEE Transactions on Information Theory, Vol. 58, Issue 5 https://doi.org/10.1109/TIT.2011.2182033	journal	May 2012
Regression Level Set Estimation Via Cost-Sensitive Classification Scott, Clayton; Davenport, Mark IEEE Transactions on Signal Processing, Vol. 55, Issue 6 https://doi.org/10.1109/TSP.2007.893758	journal	June 2007
Stochastic kriging for simulation metamodeling Ankenman, Bruce; Nelson, Barry L.; Staum, Jeremy 2008 Winter Simulation Conference (WSC) https://doi.org/10.1109/WSC.2008.4736089	conference	December 2008
Adaptive Designs of Experiments for Accurate Approximation of a Target Region Picheny, Victor; Ginsbourger, David; Roustant, Olivier Journal of Mechanical Design, Vol. 132, Issue 7 https://doi.org/10.1115/1.4001873	journal	June 2010
A Knowledge-Gradient Policy for Sequential Information Collection Frazier, Peter I.; Powell, Warren B.; Dayanik, Savas SIAM Journal on Control and Optimization, Vol. 47, Issue 5 https://doi.org/10.1137/070693424	journal	January 2008
Sequential Design for Optimal Stopping Problems Gramacy, Robert B.; Ludkovski, Michael SIAM Journal on Financial Mathematics, Vol. 6, Issue 1 https://doi.org/10.1137/140980089	journal	January 2015
Quantifying Uncertainties on Excursion Sets Under a Gaussian Random Field Prior Azzimonti, Dario; Bect, Julien; Chevalier, Clément SIAM/ASA Journal on Uncertainty Quantification, Vol. 4, Issue 1 https://doi.org/10.1137/141000749	journal	January 2016
Sequential Design for Ranking Response Surfaces Hu, Ruimeng; Ludkovski, Mike SIAM/ASA Journal on Uncertainty Quantification, Vol. 5, Issue 1 https://doi.org/10.1137/15M1045168	journal	January 2017
Bayesian Subset Simulation Bect, Julien; Li, Ling; Vazquez, Emmanuel SIAM/ASA Journal on Uncertainty Quantification, Vol. 5, Issue 1 https://doi.org/10.1137/16M1078276	journal	January 2017
Actively learning level-sets of composite functions Bryan, Brent; Schneider, Jeff Proceedings of the 25th international conference on Machine learning - ICML '08 https://doi.org/10.1145/1390156.1390167	conference	January 2008
Sequential Experiment Design for Contour Estimation From Complex Computer Codes Ranjan, Pritam; Bingham, Derek; Michailidis, George Technometrics, Vol. 50, Issue 4 https://doi.org/10.1198/004017008000000541	journal	November 2008
Adaptive Design and Analysis of Supercomputer Experiments Gramacy, Robert B.; Lee, Herbert K. H. Technometrics, Vol. 51, Issue 2 https://doi.org/10.1198/TECH.2009.0015	journal	May 2009
Kriging metamodels and experimental design for Bermudan option pricing Ludkovski, Mike Journal of Computational Finance, Vol. 22, Issue 1 https://doi.org/10.21314/JCF.2018.347	journal	January 2018
Efficient Global Reliability Analysis for Nonlinear Implicit Performance Functions Bichon, B. J.; Eldred, M. S.; Swiler, L. P. AIAA Journal, Vol. 46, Issue 10 https://doi.org/10.2514/1.34321	journal	October 2008
A sequential Bayesian algorithm to estimate a probability of failure Vazquez, Emmanuel; Bect, Julien IFAC Proceedings Volumes, Vol. 42, Issue 10 https://doi.org/10.3182/20090706-3-FR-2004.00090	journal	January 2009

Similar Records

Accelerating Noisy VQE Optimization with Gaussian Processes

Conference · 2022 · 2022 IEEE International Conference on Quantum Computing and Engineering (QCE) · OSTI ID:1961346

Muller, Juliane; Lavrijsen, Wim; Iancu, Costin; +1 more

Synchronization of noisy systems by stochastic signals

Journal Article · 1999 · Physical Review. E, Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics · OSTI ID:351864

Neiman, A; Schimansky-Geier, L; Moss, F; +3 more

Generic sequential sampling for metamodel approximations

Conference · 2003 · OSTI ID:976613

Turner, C J; Campbell, M I

Related Subjects

97 MATHEMATICS AND COMPUTING
Gaussian process
sequential updating formulas
stochastic contour-finding
student-t process

Title: Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation

Citation Formats

References (30)

Similar Records

Related Subjects