Title: Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Abstract

Deep neural networks have been successfully employed in an extensive variety of research areas, including solving partial differential equations. Despite its significant success, there are some challenges in effectively training DNN, such as avoiding overfitting in over-parameterized DNNs and accelerating the optimization in DNNs with pathological curvature. Here, we propose a Bayesian type sparse deep learning algorithm. The algorithm utilizes a set of spike-and-slab priors for the parameters in the deep neural network. The hierarchical Bayesian mixture will be trained using an adaptive empirical method. That is, one will alternatively sample from the posterior using preconditioned stochastic gradient Langevin Dynamics (PSGLD), and optimize the latent variables via stochastic approximation. The sparsity of the network is achieved while optimizing the hyperparameters with adaptive searching and penalizing. A popular SG-MCMC approach is Stochastic gradient Langevin dynamics (SGLD). However, considering the complex geometry in the model parameter space in nonconvex learning, updating parameters using a universal step size in each component as in SGLD may cause slow mixing. To address this issue, we apply a computationally manageable preconditioner in the updating rule, which provides a step-size parameter to adapt to local geometric properties. Moreover, by smoothly optimizing the hyperparameter in the preconditioning matrix,more » our proposed algorithm ensures a decreasing bias, which is introduced by ignoring the correction term in the preconditioned SGLD. According to the existing theoretical framework, we show that the proposed algorithm can asymptotically converge to the correct distribution with a controllable bias under mild conditions. Numerical tests are performed on both synthetic regression problems and learning solutions of elliptic PDE, which demonstrate the accuracy and efficiency of the present work.« less

Authors:

Wang, Yating ^[1]; Deng, Wei ^[1]; Lin, Guang ^[2]

---

+ Show Author Affiliations

1. Purdue Univ., West Lafayette, IN (United States). Dept. of Mathematics
2. Purdue Univ., West Lafayette, IN (United States). Dept. of Mathematics. School of Mechanical Engineering. Dept. of Statistics. Dept. of Earth, Atmospheric, and Planetary Sciences

Publication Date:

Wed Feb 03 00:00:00 EST 2021

Research Org.:

Purdue Univ., West Lafayette, IN (United States)

Sponsoring Org.:

USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF); US Army Research Office (ARO)

OSTI Identifier:

1853726

Alternate Identifier(s):

OSTI ID: 1809418

Grant/Contract Number:  

SC0021142; DMS-1555072; DMS-1736364; CMMI-1634832; CMMI-1560834; W911NF-15-1-0562

Resource Type:

Accepted Manuscript

Journal Name:

Journal of Computational Physics

Additional Journal Information:

Journal Volume: 432; Journal Issue: C; Journal ID: ISSN 0021-9991

Publisher:

Elsevier

Country of Publication:

United States

Language:

English

Subject:

97 MATHEMATICS AND COMPUTING; 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; computer science; physics; Bayesian sparse learning; preconditioned stochastic gradient MCMC; deep learning; deep neural network; adaptive hierarchical posterior; stochastic approximation