A scalable algorithm for the optimization of neural network architectures

Lupo Pasini, Massimiliano; Yin, Junqi; Li, Ying Wai; Eisenbach, Markus

doi:10.1016/j.parco.2021.102788

A scalable algorithm for the optimization of neural network architectures

Journal Article · Sat Apr 24 00:00:00 EDT 2021 · Parallel Computing

DOI:https://doi.org/10.1016/j.parco.2021.102788· OSTI ID:1781383

^[1]; ^[1]; ^[2]; ^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

In this work, we propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE; USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC)

Grant/Contract Number:: 89233218CNA000001; AC05-00OR22725

OSTI ID:: 1781383

Alternate ID(s):: OSTI ID: 1807280
OSTI ID: 1784469

Report Number(s):: LA-UR--21-20936

Journal Information:: Parallel Computing, Journal Name: Parallel Computing Vol. 104-105; ISSN 0167-8191

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (10)

Progressive Neural Architecture Search Liu, Chenxi; Zoph, Barret; Neumann, Maxim Computer Vision – ECCV 2018 https://doi.org/10.1007/978-3-030-01246-5_2	book	January 2018
Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position Fukushima, Kunihiko Biological Cybernetics, Vol. 36, Issue 4 https://doi.org/10.1007/BF00344251	journal	April 1980
Pruning backpropagation neural networks using modern stochastic optimisation techniques Stepniewski, Slawomir W.; Keane, Andy J. Neural Computing & Applications, Vol. 5, Issue 2 https://doi.org/10.1007/BF01501173	journal	June 1997
Optimizing Deep Feedforward Neural Network Architecture: A Tabu Search Based Approach Gupta, Tarun Kumar; Raza, Khalid Neural Processing Letters, Vol. 51, Issue 3 https://doi.org/10.1007/s11063-020-10234-7	journal	March 2020
Classification assessment methods Tharwat, Alaa Applied Computing and Informatics https://doi.org/10.1016/j.aci.2018.08.003	journal	August 2018
The perceptron: A probabilistic model for information storage and organization in the brain. Rosenblatt, F. Psychological Review, Vol. 65, Issue 6 https://doi.org/10.1037/h0042519	journal	January 1958
Exploring constructive cascade networks Treadgold, N. K.; Gedeon, T. D. IEEE Transactions on Neural Networks, Vol. 10, Issue 6 https://doi.org/10.1109/72.809079	journal	January 1999
Tuning the Structure and Parameters of a Neural Network by Using Hybrid Taguchi-Genetic Algorithm Tsai, J. -T.; Chou, J. -H.; Liu, T. -K. IEEE Transactions on Neural Networks, Vol. 17, Issue 1 https://doi.org/10.1109/TNN.2005.860885	journal	January 2006
Instance-based prediction of real-valued attributes Kibler, Dennis; Aha, David W.; Albert, Marc K. Computational Intelligence, Vol. 5, Issue 2 https://doi.org/10.1111/j.1467-8640.1989.tb00315.x	journal	February 1989
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks Hinz, Tobias; Navarro-Guerrero, Nicolás; Magg, Sven International Journal of Computational Intelligence and Applications, Vol. 17, Issue 02 https://doi.org/10.1142/S1469026818500086	journal	June 2018

Similar Records

Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensembles

Journal Article · Thu Aug 03 20:00:00 EDT 2023 · Physica. D, Nonlinear Phenomena · OSTI ID:2584756

Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach

Journal Article · Thu May 09 20:00:00 EDT 2024 · Mathematics · OSTI ID:2477212

Related Subjects

97 MATHEMATICS AND COMPUTING
adaptive algorithms
deep learning
greedy constructive algorithms
hyperparameter optimization
neural network architecture
random search

A scalable algorithm for the optimization of neural network architectures

Citation Formats

References (10)

Similar Records

Related Subjects