Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

Luo, Hengrui; Cho, Younghyun; Demmel, James W.; Li, Xiaoye S.; Liu, Yang

doi:10.1080/10618600.2024.2308216

Title: Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

Journal Article · 07 March 2024 · Journal of Computational and Graphical Statistics

DOI: https://doi.org/10.1080/10618600.2024.2308216 · OSTI ID:2426857

^[1]; Cho, Younghyun ^[2]; Demmel, James W. ^[2]; Li, Xiaoye S. ^[1];

^[1]

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Univ. of California, Berkeley, CA (United States)

Herein this article presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture’s integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 2426857

Journal Information:: Journal of Computational and Graphical Statistics, Journal Name: Journal of Computational and Graphical Statistics Journal Issue: 3 Vol. 33; ISSN 1061-8600

Publisher:: Taylor & FrancisCopyright Statement

Country of Publication:: United States

Language:: English

References (29)

Bandit Based Monte-Carlo Planning Kocsis, Levente; Szepesvári, Csaba Lecture Notes in Computer Science https://doi.org/10.1007/11871842_29	book	January 2006
TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning Olson, Randal S.; Moore, Jason H. Automated Machine Learning https://doi.org/10.1007/978-3-030-05318-5_8	book	January 2019
Sequential Model-Based Optimization for General Algorithm Configuration Hutter, Frank; Hoos, Holger H.; Leyton-Brown, Kevin Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-25566-3_40	book	January 2011
Black-box combinatorial optimization using models with integer-valued minima Bliek, Laurens; Verwer, Sicco; de Weerdt, Mathijs Annals of Mathematics and Artificial Intelligence, Vol. 89, Issue 7 https://doi.org/10.1007/s10472-020-09712-4	journal	September 2020
Similarity encoding for learning with dirty categorical variables Cerda, Patricio; Varoquaux, Gaël; Kégl, Balázs Machine Learning, Vol. 107, Issue 8-10 https://doi.org/10.1007/s10994-018-5724-2	journal	June 2018
Progressive sampling-based Bayesian optimization for efficient and automatic machine learning model selection Zeng, Xueqiang; Luo, Gang Health Information Science and Systems, Vol. 5, Issue 1 https://doi.org/10.1007/s13755-017-0023-z	journal	September 2017
Hedonic housing prices and the demand for clean air Harrison, David; Rubinfeld, Daniel L. Journal of Environmental Economics and Management, Vol. 5, Issue 1 https://doi.org/10.1016/0095-0696(78)90006-2	journal	March 1978
Dealing with categorical and integer-valued variables in Bayesian Optimization with Gaussian processes Garrido-Merchán, Eduardo C.; Hernández-Lobato, Daniel Neurocomputing, Vol. 380 https://doi.org/10.1016/j.neucom.2019.11.004	journal	March 2020
Efficient Global Optimization of Expensive Black-Box Functions Jones, Donald R.; Schonlau, Matthias; Welch, William J. Journal of Global Optimization, Vol. 13, Issue 4, p. 455-492 https://doi.org/10.1023/A:1008306431147	journal	January 1998
Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables Zhang, Yichi; Apley, Daniel W.; Chen, Wei Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-60652-9	journal	March 2020
Additive Gaussian Process for Computer Models With Qualitative and Quantitative Factors Deng, X.; Lin, C. Devon; Liu, K. -W. Technometrics, Vol. 59, Issue 3 https://doi.org/10.1080/00401706.2016.1211554	journal	April 2017
A Latent Variable Approach to Gaussian Process Modeling with Qualitative and Quantitative Factors Zhang, Yichi; Tao, Siyu; Chen, Wei Technometrics, Vol. 62, Issue 3 https://doi.org/10.1080/00401706.2019.1638834	journal	August 2019
On Measuring and Correcting the Effects of Data Mining and Model Selection Ye, Jianming Journal of the American Statistical Association, Vol. 93, Issue 441 https://doi.org/10.1080/01621459.1998.10474094	journal	March 1998
3d-Hst Wfc3-Selected Photometric Catalogs in the five Candels/3d-Hst Fields: Photometry, Photometric Redshifts, and Stellar Masses Skelton, Rosalind E.; Whitaker, Katherine E.; Momcheva, Ivelina G. The Astrophysical Journal Supplement Series, Vol. 214, Issue 2 https://doi.org/10.1088/0067-0049/214/2/24	journal	October 2014
Taking the Human Out of the Loop: A Review of Bayesian Optimization Shahriari, Bobak; Swersky, Kevin; Wang, Ziyu Proceedings of the IEEE, Vol. 104, Issue 1 https://doi.org/10.1109/JPROC.2015.2494218	journal	January 2016
Autotuning in High-Performance Computing Applications Balaprakash, Prasanna; Dongarra, Jack; Gamblin, Todd Proceedings of the IEEE, Vol. 106, Issue 11 https://doi.org/10.1109/JPROC.2018.2841200	journal	November 2018
Bayesian Optimization for auto-tuning GPU kernels Willemsen, Floris-Jan; van Nieuwpoort, Rob; van Werkhoven, Ben 2021 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) https://doi.org/10.1109/PMBS54543.2021.00017	conference	November 2021
The Determination of the Order of an Autoregression Hannan, E. J.; Quinn, B. G. Journal of the Royal Statistical Society Series B: Statistical Methodology, Vol. 41, Issue 2 https://doi.org/10.1111/j.2517-6161.1979.tb01072.x	journal	January 1979
An Efficient Multicore Implementation of a Novel HSS-Structured Multifrontal Solver Using Randomized Sampling Ghysels, Pieter; Li, Xiaoye S.; Rouet, François-Henry SIAM Journal on Scientific Computing, Vol. 38, Issue 5 https://doi.org/10.1137/15M1010117	journal	January 2016
EzGP: Easy-to-Interpret Gaussian Process Models for Computer Experiments with Both Quantitative and Qualitative Factors Xiao, Qian; Mandal, Abhyuday; Lin, C. Devon SIAM/ASA Journal on Uncertainty Quantification, Vol. 9, Issue 2 https://doi.org/10.1137/19M1288462	journal	January 2021
Scaling Multi-Armed Bandit Algorithms Fouché, Edouard; Komiyama, Junpei; Böhm, Klemens Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3292500.3330862	conference	July 2019
Recurrent Kernel Machines: Computing with Infinite Echo State Networks Hermans, Michiel; Schrauwen, Benjamin Neural Computation, Vol. 24, Issue 1 https://doi.org/10.1162/NECO_a_00200	journal	January 2012
Surrogates Gramacy, Robert B. https://doi.org/10.1201/9780367815493	book	January 2020
BART: Bayesian additive regression trees Chipman, Hugh A.; George, Edward I.; McCulloch, Robert E. The Annals of Applied Statistics, Vol. 4, Issue 1 https://doi.org/10.1214/09-AOAS285	journal	March 2010
Multivariate Adaptive Regression Splines Friedman, Jerome H. The Annals of Statistics, Vol. 19, Issue 1 https://doi.org/10.1214/aos/1176347963	journal	March 1991
Bayesian Optimization for Categorical and Category-Specific Continuous Inputs Nguyen, Dang; Gupta, Sunil; Rana, Santu Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, Issue 04 https://doi.org/10.1609/aaai.v34i04.5971	journal	April 2020
Automated Machine Learning with Monte-Carlo Tree Search Rakotoarison, Herilalaina; Schoenauer, Marc; Sebag, Michèle Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2019/457	conference	August 2019
Automatic Kernel Selection for Gaussian Processes Regression with Approximate Bayesian Computation and Sequential Monte Carlo Abdessalem, Anis Ben; Dervilis, Nikolaos; Wagg, David J. Frontiers in Built Environment, Vol. 3 https://doi.org/10.3389/fbuil.2017.00052	journal	August 2017
Additive Gaussian Process for Computer Models With Qualitative and Quantitative Factors Deng, X.; Lin, C. Devon; Liu, K. -W. Taylor & Francis https://doi.org/10.6084/m9.figshare.5395249	dataset	January 2017

Similar Records

Physics makes the difference: Bayesian optimization and active learning via augmented Gaussian process

Journal Article · 2022 · Machine Learning: Science and Technology · OSTI ID:1843878

Ziatdinov, Maxim A.; Ghosh, Ayana; Kalinin, Sergei V.

Descriptor Aided Bayesian Optimization for Many-Level Qualitative Variables With Materials Design Applications

Journal Article · 2022 · Journal of Mechanical Design · OSTI ID:2422335

Iyer, Akshay; Yerramilli, Suraj; Rondinelli, James M.; +2 more

Uncertainty-aware mixed-variable machine learning for materials design

Journal Article · 2022 · Scientific Reports · OSTI ID:1898694

Zhang, Hengrui; Chen, Wei; Iyer, Akshay; +2 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Gaussian processes
Monte Carlo tree search
categorical variables
online kernel selection

Title: Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

Citation Formats

References (29)

Similar Records

Related Subjects