Layer-Parallel Training of Deep Residual Neural Networks

Journal Article · Thu Feb 06 00:00:00 EST 2020 · SIAM Journal on Mathematics of Data Science

DOI:https://doi.org/10.1137/19M1247620· OSTI ID:1618082

Günther, Stefanie ^[1]; Ruthotto, Lars ^[2]; Schroder, Jacob B. ^[3]; Cyr, Eric C. ^[4]; Gauger, Nicolas R. ^[1]

---

1. Univ. of Kaiserslautern (Germany)
2. Emory Univ., Atlanta, GA (United States)
3. Univ. of New Mexico, Albuquerque, NM (United States)
4. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers with a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Finally, using numerical examples from supervised classification, we demonstrate that the new approach achieves a training performance similar to that of traditional methods, but enables layer-parallelism and thus provides speedup over layer-serial methods through greater concurrency.

• Research Organization: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
• Sponsoring Organization: National Science Foundation (NSF); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• Grant/Contract Number: AC04-94AL85000; NA0003525
• OSTI ID: 1618082
• Report Number(s): SAND--2019-12660J; 680497
• Journal Information: SIAM Journal on Mathematics of Data Science, Journal Name: SIAM Journal on Mathematics of Data Science Journal Issue: 1 Vol. 2; ISSN 2577-0187
• Publisher: Society for Industrial and Applied Mathematics (SIAM)
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
deep learning
layer-parallelization
optimal control
parallel-in-time
residual networks
simultaneous optimization
supervised learning