Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Wang, Bao; Ye, Qiang

doi:10.1109/tnnls.2023.3255783

Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Journal Article · Fri Mar 24 00:00:00 EDT 2023 · IEEE Transactions on Neural Networks and Learning Systems

DOI:https://doi.org/10.1109/tnnls.2023.3255783· OSTI ID:2280651

^[1]; Ye, Qiang ^[2]

University of Utah, Salt Lake City, UT (United States); Department of Mathematics
University of Kentucky, Lexington, KY (United States)

Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparame- ter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For example, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.

View Accepted Manuscript (DOE)

Research Organization:: University of Utah, Salt Lake City, UT (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF)

Grant/Contract Number:: SC0021142; SC0023490

OSTI ID:: 2280651

Journal Information:: IEEE Transactions on Neural Networks and Learning Systems, Journal Name: IEEE Transactions on Neural Networks and Learning Systems Journal Issue: 9 Vol. 35; ISSN 2162-237X

Publisher:: IEEE Computational Intelligence SocietyCopyright Statement

Country of Publication:: United States

Language:: English

References (27)

Identity Mappings in Deep Residual Networks He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing Computer Vision – ECCV 2016 https://doi.org/10.1007/978-3-319-46493-0_38	book	January 2016
Some convergence properties of the conjugate gradient method Powell, M. J. D. Mathematical Programming, Vol. 11, Issue 1 https://doi.org/10.1007/BF01580369	journal	December 1976
First-order methods of smooth convex optimization with inexact oracle Devolder, Olivier; Glineur, François; Nesterov, Yurii Mathematical Programming, Vol. 146, Issue 1-2 https://doi.org/10.1007/s10107-013-0677-5	journal	June 2013
Adaptive Restart for Accelerated Gradient Schemes O’Donoghue, Brendan; Candès, Emmanuel Foundations of Computational Mathematics, Vol. 15, Issue 3 https://doi.org/10.1007/s10208-013-9150-3	journal	July 2013
ImageNet Large Scale Visual Recognition Challenge Russakovsky, Olga; Deng, Jia; Su, Hao International Journal of Computer Vision, Vol. 115, Issue 3 https://doi.org/10.1007/s11263-015-0816-y	journal	April 2015
Some methods of speeding up the convergence of iteration methods Polyak, B. T. USSR Computational Mathematics and Mathematical Physics, Vol. 4, Issue 5 https://doi.org/10.1016/0041-5553(64)90137-5	journal	January 1964
Optimal methods of smooth convex minimization Nemirovskii, A. S.; Nesterov, Yu. E. USSR Computational Mathematics and Mathematical Physics, Vol. 25, Issue 2 https://doi.org/10.1016/0041-5553(85)90100-4	journal	January 1985
A scaled conjugate gradient algorithm for fast supervised learning Møller, Martin Fodslette Neural Networks, Vol. 6, Issue 4 https://doi.org/10.1016/S0893-6080(05)80056-5	journal	January 1993
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Note sur la convergence de méthodes de directions conjuguées Polak, E.; Ribiere, G. Revue française d'informatique et de recherche opérationnelle. Série rouge, Vol. 3, Issue 16 https://doi.org/10.1051/m2an/196903R100351	journal	January 1969
Analysis of the finite precision bi-conjugate gradient algorithm for nonsymmetric linear systems Tong, Charles H.; Ye, Qiang Mathematics of Computation, Vol. 69, Issue 232 https://doi.org/10.1090/S0025-5718-99-01171-0	journal	August 1999
Function minimization by conjugate gradients Fletcher, R. The Computer Journal, Vol. 7, Issue 2 https://doi.org/10.1093/comjnl/7.2.149	journal	February 1964
Descent Property and Global Convergence of the Fletcher—Reeves Method with Inexact Line Search Al-Baali, M. IMA Journal of Numerical Analysis, Vol. 5, Issue 1 https://doi.org/10.1093/imanum/5.1.121	journal	January 1985
Monotonicity and restart in fast gradient methods Giselsson, Pontus; Boyd, Stephen 53rd IEEE Conference on Decision and Control https://doi.org/10.1109/CDC.2014.7040179	conference	December 2014
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Advances in optimizing recurrent networks Bengio, Yoshua; Boulanger-Lewandowski, Nicolas; Pascanu, Razvan 2013 IEEE International Conference on Acoustics, Speech and Signal Processing https://doi.org/10.1109/ICASSP.2013.6639349	conference	May 2013
CAD: Scale Invariant Framework for Real-Time Object Detection Zhou, Huajun; Li, Zechao; Ning, Chengcheng 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) https://doi.org/10.1109/ICCVW.2017.95	conference	October 2017
Inception Single Shot MultiBox Detector for object detection No authors listed 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) https://doi.org/10.1109/ICMEW.2017.8026312	conference	July 2017
Towards Evaluating the Robustness of Neural Networks Carlini, Nicholas; Wagner, David 2017 IEEE Symposium on Security and Privacy (SP) https://doi.org/10.1109/SP.2017.49	conference	May 2017
A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search Hager, William W.; Zhang, Hongchao SIAM Journal on Optimization, Vol. 16, Issue 1 https://doi.org/10.1137/030601880	journal	January 2005
Global Convergence Properties of Conjugate Gradient Methods for Optimization Gilbert, Jean Charles; Nocedal, Jorge SIAM Journal on Optimization, Vol. 2, Issue 1 https://doi.org/10.1137/0802003	journal	February 1992
Iterative Methods for Sparse Linear Systems Saad, Yousef https://doi.org/10.1137/1.9780898718003	book	January 2003
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
On the Generalized Lanczos Trust-Region Method Zhang, Lei-Hong; Shen, Chungen; Li, Ren-Cang SIAM Journal on Optimization, Vol. 27, Issue 3 https://doi.org/10.1137/16M1095056	journal	January 2017
Sharpness, Restart, and Acceleration Roulet, Vincent; d'Aspremont, Alexandre SIAM Journal on Optimization, Vol. 30, Issue 1 https://doi.org/10.1137/18M1224568	journal	January 2020
Inexact Preconditioned Conjugate Gradient Method with Inner-Outer Iteration Golub, Gene H.; Ye, Qiang SIAM Journal on Scientific Computing, Vol. 21, Issue 4 https://doi.org/10.1137/S1064827597323415	journal	January 1999
A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property Dai, Y. H.; Yuan, Y. SIAM Journal on Optimization, Vol. 10, Issue 1 https://doi.org/10.1137/s1052623497318992	journal	January 1999

Similar Records

XploreNAS: Explore Adversarially Robust and Hardware-efficient Neural Architectures for Non-ideal Xbars

Journal Article · Mon Jul 24 00:00:00 EDT 2023 · ACM Transactions on Embedded Computing Systems · OSTI ID:2422212

Scalable Second Order Optimization for Machine Learning

Technical Report · Sun May 01 00:00:00 EDT 2022 · OSTI ID:1984057

Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Journal Article · Tue Feb 02 23:00:00 EST 2021 · Journal of Computational Physics · OSTI ID:1853726

Related Subjects

97 MATHEMATICS AND COMPUTING
Adaptive momentum
deep learning
image classification
nonlinear conjugate gradient (NCG)

Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Citation Formats

References (27)

Similar Records

Related Subjects