HyperSpace: Distributed Bayesian Hyperparameter Optimization

doi:10.1109/CAHPC.2018.8645954

Title: HyperSpace: Distributed Bayesian Hyperparameter Optimization

Full Record
Other Related Research

Abstract

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.

Authors:

Young, M. Todd ^[1]; Hinkle, Jacob ^[1]; Ramanathan, Arvind ^[1]; Kannan, Ramakrishnan ^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: 2018-09-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1556107

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Proceedings (Symposium on Computer Architecture and High Performance Computing)

Additional Journal Information:: Journal Name: Proceedings (Symposium on Computer Architecture and High Performance Computing); Conference: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), Lyon (France), 24-27 Sept. 2018; Journal ID: ISSN 1550-6533

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States: N. p., 2018. 
        Web.  doi:10.1109/CAHPC.2018.8645954.

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, & Kannan, Ramakrishnan. HyperSpace: Distributed Bayesian Hyperparameter Optimization.  United States.  doi:10.1109/CAHPC.2018.8645954.

Copy to clipboard


                    Young, M. Todd, Hinkle, Jacob, Ramanathan, Arvind, and Kannan, Ramakrishnan. Sat .  
        "HyperSpace: Distributed Bayesian Hyperparameter Optimization".  United States.  doi:10.1109/CAHPC.2018.8645954.  https://www.osti.gov/servlets/purl/1556107.

Copy to clipboard


                    
@article{osti_1556107,

  title        = {HyperSpace: Distributed Bayesian Hyperparameter Optimization},

  author       = {Young, M. Todd and Hinkle, Jacob and Ramanathan, Arvind and Kannan, Ramakrishnan},

  abstractNote = {As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show that it is possible to learn the dependencies between model hyperparameters through the optimization process. By partitioning large search spaces and running many optimization procedures in parallel, we also show that it is possible to discover families of good hyperparameter settings over a variety of models including unsupervised clustering, regression, and classification tasks.},

  doi          = {10.1109/CAHPC.2018.8645954},

  journal      = {Proceedings (Symposium on Computer Architecture and High Performance Computing)},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {9}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

DOI: 10.1109/CAHPC.2018.8645954

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

HyperSpace: Distributed Bayesian Hyperparameter Optimization

Conference Young, M. Todd ; Hinkle, Jacob ; Ramanathan, Arvind ; ... - 2018 30TH INTERNATIONAL SYMPOSIUM ON COMPUTER ARCHITECTURE AND HIGH PERFORMANCE COMPUTING (SBAC-PAD 2018)

As machine learning models continue to increase in complexity, so does the potential number of free model parameters commonly known as hyperparameters. While there has been considerable progress toward finding optimal configurations of these hyperparameters, many optimization procedures are treated as black boxes. We believe optimization methods should not only return a set of optimized hyperparameters, but also give insight into the effects of model hyperparameter settings. To this end, we present HyperSpace, a parallel implementation of Bayesian sequential model-based optimization. HyperSpace leverages high performance computing (HPC) resources to better understand unknown, potentially non-convex hyperparameter search spaces. We show thatmore »« less
DOI: 10.1109/CAHPC.2018.8645954
HyperSpace

Software Ramanathan, Arvind ; Hinkle, Jacob D. ; Kannan, Ramakrishnan ; ...

Machine learning (ML) models often contain numerous hyperparameters, free parameters that must be set before the models can be trained. As the number of model hyperparameters increases, their optimization becomes significantly more challenging as we face a combinatorial increase in potential model configurations. Similarly, there is an increased chance that our models’ hyperparameters interact in complex ways. HyperSpace allows a user to optimize complex machine learning algorithms based on their hyperparameters. HyperSpace works by parallelizing parameter search spaces, running Bayesian model based optimization (SMBO) over each of these spaces in parallel. It was designed to be as minimally invasive asmore »« less
DOI: 10.5281/zenodo.1401479

View Software
PABO: Pseudo Agent-Based Multi-Objective Bayesian Hyperparameter Optimization for Efficient Neural Accelerator Design

Conference Parsa, Maryam ; Ankiit, Aayush ; Ziabari, Amir ; ...

The ever increasing computational cost of Deep Neural Networks (DNN) and the demand for energy efficient hardware for DNN acceleration has made accuracy and hardware cost co-optimization for DNNs tremendously important, especially for edge devices. Owing to the large parameter space and cost of evaluating each parameter in the search space, manually tuning of DNN hyperparameters is impractical. Automatic joint DNN and hardware hyperparameter optimization is indispensable for such problems. Bayesian optimization-based approaches have shown promising results for hyperparameter optimization of DNNs. However, most of these techniques have been developed without considering the underlying hardware, thereby leading to inefficient designs.more »« less
DOI: 10.1109/ICCAD45719.2019.8942046

Full Text Available
Bayesian Multi-objective Hyperparameter Optimization for Accurate, Fast, and Efficient Neural Network Accelerator Design

Journal Article Parsa, Maryam ; Mitchell, John P. ; Schuman, Catherine D. ; ... - Frontiers in Neuroscience (Online)

In resource-constrained environments, such as low-power edge devices and smart sensors, deploying a fast, compact, and accurate intelligent system with minimum energy is indispensable. Embedding intelligence can be achieved using neural networks on neuromorphic hardware. Designing such networks would require determining several inherent hyperparameters. A key challenge is to find the optimum set of hyperparameters that might belong to the input/output encoding modules, the neural network itself, the application, or the underlying hardware. In this work, we present a hierarchical pseudo agent-based multi-objective Bayesian hyperparameter optimization framework (both software and hardware) that not only maximizes the performance of the network,more »« less
DOI: 10.3389/fnins.2020.00667
Bayesian-based Hyperparameter Optimization for Spiking Neuromorphic Systems

Conference Parsa, Maryam ; Mitchell, Parker ; Schuman, Catherine ; ...

Designing a neuromorphic computing system involves selection of several hyperparameters that not only affect the accuracy of the framework, but also the energy efficiency and speed of inference and training. These hyperparameters might be inherent to the training of the spiking neural network (SNN), the input/output encoding of the real-world data to spikes, or the underlying neuromorphic hardware. In this work, we present a Bayesian-based hyperparameter optimization approach for spiking neuromorphic systems, and we show how this optimization framework can lead to significant improvement in designing accurate neuromorphic computing systems. In particular, we show that this hyperparameter optimization approach canmore »« less
DOI: 10.1109/BigData47090.2019.9006383

Full Text Available

Similar Records