Train Like a (Var)Pro: Efficient Training of Neural Networks with Variable Projection

Journal Article · Tue Oct 05 00:00:00 EDT 2021 · SIAM Journal on Mathematics of Data Science

DOI:https://doi.org/10.1137/20m1359511· OSTI ID:1834344

Newman, Elizabeth ^[1]; Ruthotto, Lars ^[1]; Hart, Joseph ^[2]; van Bloemen Waanders, Bart ^[2]

---

1. Emory Univ., Atlanta, GA (United States)
2. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Deep neural networks (DNNs) have achieved state-of-the-art performance across a variety of traditional machine learning tasks, e.g., speech recognition, image classification, and segmentation. The ability of DNNs to efficiently approximate high-dimensional functions has also motivated their use in scientific applications, e.g., to solve partial differential equations and to generate surrogate models. In this paper, we consider the supervised training of DNNs, which arises in many of the above applications. We focus on the central problem of optimizing the weights of the given DNN such that it accurately approximates the relation between observed input and target data. Devising effective solvers for this optimization problem is notoriously challenging due to the large number of weights, nonconvexity, data sparsity, and nontrivial choice of hyperparameters. To solve the optimization problem more efficiently, we propose the use of variable projection (VarPro), a method originally designed for separable nonlinear least-squares problems. Our main contribution is the Gauss--Newton VarPro method (GNvpro) that extends the reach of the VarPro idea to nonquadratic objective functions, most notably cross-entropy loss functions arising in classification. These extensions make GNvpro applicable to all training problems that involve a DNN whose last layer is an affine mapping, which is common in many state-of-the-art architectures. In our four numerical experiments from surrogate modeling, segmentation, and classification, GNvpro solves the optimization problem more efficiently than commonly used stochastic gradient descent (SGD) schemes. Finally, GNvpro finds solutions that generalize well, and in all but one example better than well-tuned SGD methods, to unseen data points.

• Research Organization: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA)
• Grant/Contract Number: NA0003525
• OSTI ID: 1834344
• Report Number(s): SAND--2020-8481J; 689974
• Journal Information: SIAM Journal on Mathematics of Data Science, Journal Name: SIAM Journal on Mathematics of Data Science Journal Issue: 4 Vol. 3; ISSN 2577-0187
• Publisher: Society for Industrial and Applied Mathematics (SIAM)
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
PDE surrogate modeling
deep learning
hyperspectral segmentation
neural networks
numerical optimization
variable projection