HyperSpace: Distributed Bayesian Hyperparameter Optimization

Young, M. Todd; Hinkle, Jacob; Ramanathan, Arvind; Kannan, Ramakrishnan

doi:10.1109/CAHPC.2018.8645954

Title: HyperSpace: Distributed Bayesian Hyperparameter Optimization

Full Record
Other Related Research

Abstract

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.

Authors:

Young, M. Todd ^[1]; Hinkle, Jacob ^[1]; Ramanathan, Arvind ^[1]; Kannan, Ramakrishnan ^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: 2018-09-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1556107

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Proceedings (Symposium on Computer Architecture and High Performance Computing)

Additional Journal Information:: Journal Name: Proceedings (Symposium on Computer Architecture and High Performance Computing); Conference: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), Lyon (France), 24-27 Sept. 2018; Journal ID: ISSN 1550-6533

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States: N. p., 2018. 
Web.  doi:10.1109/CAHPC.2018.8645954.

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, & Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States.  https://doi.org/10.1109/CAHPC.2018.8645954

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. Sat .  
"HyperSpace: Distributed Bayesian Hyperparameter Optimization".  United States.  https://doi.org/10.1109/CAHPC.2018.8645954.  https://www.osti.gov/servlets/purl/1556107.

Copy to clipboard


                    
@article{osti_1556107,

  title        = {HyperSpace: Distributed Bayesian Hyperparameter Optimization},

  author       = {Young, M. Todd and Hinkle, Jacob and Ramanathan, Arvind and Kannan, Ramakrishnan},

  abstractNote = {As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.},

  doi          = {10.1109/CAHPC.2018.8645954},

  journal      = {Proceedings (Symposium on Computer Architecture and High Performance Computing)},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {9}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/CAHPC.2018.8645954

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

HyperSpace: Distributed Bayesian Hyperparameter Optimization

Conference Young, M. Todd ; Hinkle, Jacob ; Ramanathan, Arvind ; ... - 2018 30TH INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE AND HIGH PERFORMANCE COMPUTING (SBAC-PAD 2018)

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show thatmore »« less
https://doi.org/10.1109/CAHPC.2018.8645954
HyperSpace

Software Ramanathan, Arvind ; Hinkle, Jacob D. ; Kannan, Ramakrishnan ; ...

Machine learning (ML) models often contain numerous hyperparameters, free parameters that must be set before the models can be trained. As the number of model hyperparameters increases, their optimization becomes significantly more challenging as we face a combinatorial increase in potential model configurations. Similarly, there is an increased chance that our models’ hyperparameters interact in complex ways. HyperSpace allows a user to optimize complex machine learning algorithms based on their hyperparameters. HyperSpace works by parallelizing parameter search spaces, running Bayesian model based optimization (SMBO) over each of these spaces in parallel. It was designed to be as minimally invasive asmore »« less
https://doi.org/10.5281/zenodo.1401479

View Software
DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks

Conference Balaprakash, Prasanna ; Salim, Michael ; Uram, Thomas D. ; ...

Hyperparameters employed by deep learning (DL) methods play a substantial role in the performance and reliability of these methods in practice. Unfortunately, finding performance-optimizing hyperparameter settings is a notoriously difficult task. Hyperparameter search methods typically have limited production-strength implementations or do not target scalability within a highly parallel machine, portability across different machines, experimental comparison between different methods, and tighter integration with workflow systems. In this paper, we present DeepHyper, a Python package that provides a common interface for the implementation and study of scalable hyperparameter search methods. It adopts the Balsam workflow system to hide the complexities of runningmore »« less
https://doi.org/10.1109/HiPC.2018.00014
DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks

Conference Balaprakash, Prasanna ; Salim, Michael ; Uram, Thomas D. ; ...

Hyperparameters employed by deep learning (DL) methods play a substantial role in the performance and reliability of these methods in practice. Unfortunately, finding performance-optimizing hyperparameter settings is a notoriously difficult task. Hyperparameter search methods typically have limited production-strength implementations or do not target scalability within a highly parallel machine, portability across different machines, experimental comparison between different methods, and tighter integration with workflow systems. In this paper, we present DeepHyper, a Python package that provides a common interface for the implementation and study of scalable hyperparameter search methods. It adopts the Balsam workflow system to hide the complexities of runningmore »« less
https://doi.org/10.1109/HiPC.2018.00014
Distributed Bayesian optimization of deep reinforcement learning algorithms

Journal Article Young, M. Todd ; Hinkle, Jacob D. ; Kannan, Ramakrishnan ; ... - Journal of Parallel and Distributed Computing

Significant strides have been made in supervised learning settings thanks to the successful application of deep learning. Now, recent work has brought the techniques of deep learning to bear on sequential decision processes in the area of deep reinforcement learning (DRL). Currently, little is known regarding hyperparameter optimization for DRL algorithms. Given that DRL algorithms are computationally intensive to train, and are known to be sample inefficient, optimizing model hyperparameters for DRL presents significant challenges to established techniques. We provide an open source, distributed Bayesian model-based optimization algorithm, HyperSpace, and show that it consistently outperforms standard hyperparameter optimization techniques acrossmore » three DRL algorithms.« less
https://doi.org/10.1016/j.jpdc.2019.07.008

Similar Records