HyperSpace: Distributed Bayesian Hyperparameter Optimization

doi:10.1109/CAHPC.2018.8645954

Title: HyperSpace: Distributed Bayesian Hyperparameter Optimization

Full Record
Other Related Research

Abstract

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.

Authors:

Young, M. Todd ^[1]; Hinkle, Jacob ^[1]; Ramanathan, Arvind ^[1]; Kannan, Ramakrishnan ^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: 2018-09-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1556107

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Proceedings (Symposium on Computer Architecture and High Performance Computing)

Additional Journal Information:: Journal Name: Proceedings (Symposium on Computer Architecture and High Performance Computing); Conference: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), Lyon (France), 24-27 Sept. 2018; Journal ID: ISSN 1550-6533

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States: N. p., 2018. 
        Web.  doi:10.1109/CAHPC.2018.8645954.

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, & Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States.  doi:10.1109/CAHPC.2018.8645954.

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. Sat .  
        "HyperSpace: Distributed Bayesian Hyperparameter Optimization".  United States.  doi:10.1109/CAHPC.2018.8645954.  https://www.osti.gov/servlets/purl/1556107.

Copy to clipboard


                    
@article{osti_1556107,

  title        = {HyperSpace: Distributed Bayesian Hyperparameter Optimization},

  author       = {Young, M. Todd and Hinkle, Jacob and Ramanathan, Arvind and Kannan, Ramakrishnan},

  abstractNote = {As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.},

  doi          = {10.1109/CAHPC.2018.8645954},

  journal      = {Proceedings (Symposium on Computer Architecture and High Performance Computing)},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {9}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

DOI: 10.1109/CAHPC.2018.8645954

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

HyperSpace: Distributed Bayesian Hyperparameter Optimization

Conference Young, M. Todd ; Hinkle, Jacob ; Ramanathan, Arvind ; ... - 2018 30TH INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE AND HIGH PERFORMANCE COMPUTING (SBAC-PAD 2018)

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show thatmore »« less
DOI: 10.1109/CAHPC.2018.8645954
HyperSpace

Software Ramanathan, Arvind ; Hinkle, Jacob D ; Kannan, Ramakrishnan ; ...

Machine learning (ML) models often contain numerous hyperparameters, free parameters that must be set before the models can be trained. As the number of model hyperparameters increases, their optimization becomes significantly more challenging as we face a combinatorial increase in potential model configurations. Similarly, there is an increased chance that our models’ hyperparameters interact in complex ways. HyperSpace allows a user to optimize complex machine learning algorithms based on their hyperparameters. HyperSpace works by parallelizing parameter search spaces, running Bayesian model based optimization (SMBO) over each of these spaces in parallel. It was designed to be as minimally invasive asmore »« less
View Software
Unsupervised Group Discovery and LInk Prediction in Relational Datasets: a nonparametric Bayesian approach

Technical Report Koutsourelakis, P

Clustering represents one of the most common statistical procedures and a standard tool for pattern discovery and dimension reduction. Most often the objects to be clustered are described by a set of measurements or observables e.g. the coordinates of the vectors, the attributes of people. In a lot of cases however the available observations appear in the form of links or connections (e.g. communication or transaction networks). This data contains valuable information that can in general be exploited in order to discover groups and better understand the structure of the dataset. Since in most real-world datasets, several of these linksmore »« less
DOI: 10.2172/908093

Full Text Available
Fully Bayesian estimation of Gibbs hyperparameters for emission computed tomography data

Journal Article Higdon, D.M. ; Johnson, V.E. ; Bowsher, J.E. ; ... - IEEE Transactions on Medical Imaging

In recent years, many investigators have proposed Gibbs prior models to regularize images reconstructed from emission computed tomography data. Unfortunately, hyperparameters used to specify Gibbs priors can greatly influence the degree of regularity imposed by such priors and, as a result, numerous procedures have been proposed to estimate hyperparameter values from observed image data. Many of these procedures attempt to maximize the joint posterior distribution on the image scene. To implement these methods, approximations to the joint posterior densities are required, because the dependence of the Gibbs partition function on the hyperparameter values is unknown. In this paper, the authorsmore »« less
DOI: 10.1109/42.640741
Order priors for Bayesian network discovery with an application to malware phylogeny

Journal Article Oyen, Diane ; Anderson, Blake ; Sentz, Kari ; ... - Statistical Analysis and Data Mining

Here, Bayesian networks have been used extensively to model and discover dependency relationships among sets of random variables. We learn Bayesian network structure with a combination of human knowledge about the partial ordering of variables and statistical inference of conditional dependencies from observed data. Our approach leverages complementary information from human knowledge and inference from observed data to produce networks that reflect human beliefs about the system as well as to fit the observed data. Applying prior beliefs about partial orderings of variables is an approach distinctly different from existing methods that incorporate prior beliefs about direct dependencies (or edges)more »« less
DOI: 10.1002/sam.11364

Full Text Available

Similar Records