Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

Wang, Bao; Zou, Difan; Gu, Quanquan; Osher, Stanley J.

doi:10.1137/19m1294356

Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

Journal Article · Mon Jan 04 00:00:00 EST 2021 · SIAM Journal on Scientific Computing

DOI:https://doi.org/10.1137/19m1294356· OSTI ID:1866812

^[1]; Zou, Difan ^[2]; Gu, Quanquan ^[2]; Osher, Stanley J. ^[2]

Univ. of California, Los Angeles, CA (United States); University of Utah
Univ. of California, Los Angeles, CA (United States)

As an important Markov chain Monte Carlo (MCMC) method, the stochastic gradient Langevin dynamics (SGLD) algorithm has achieved great success in Bayesian learning and posterior sampling. Furthermore, SGLD typically suffers from a slow convergence rate due to its large variance caused by the stochastic gradient. In order to alleviate these drawbacks, we leverage the recently developed Laplacian smoothing technique and propose a Laplacian smoothing stochastic gradient Langevin dynamics (LS-SGLD) algorithm. We prove that for sampling from both log-concave and non-log-concave densities, LS-SGLD achieves strictly smaller discretization error in 2-Wasserstein distance, although its mixing rate can be slightly slower. Experiments on both synthetic and real datasets verify our theoretical results and demonstrate the superior performance of LS-SGLD on different machine learning tasks including posterior sampling, Bayesian logistic regression, and training Bayesian convolutional neural networks.

Research Organization:: Univ. of Utah, Salt Lake City, UT (United States)

Sponsoring Organization:: Air Force Research Laboratory; National Science Foundation; Office of Naval Research; USDOE

Grant/Contract Number:: SC0021142

OSTI ID:: 1866812

Alternate ID(s):: OSTI ID: 1853730

Journal Information:: SIAM Journal on Scientific Computing, Journal Name: SIAM Journal on Scientific Computing Journal Issue: 1 Vol. 43; ISSN 1064-8275

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (12)

Mimicking the one-dimensional marginal distributions of processes having an ito differential Gyöngy, I. Probability Theory and Related Fields, Vol. 71, Issue 4 https://doi.org/10.1007/bf00699039	journal	December 1986
Diffusion for Global Optimization in $\mathbb{R}^n $ Chiang, Tzuu-Shuh; Hwang, Chii-Ruey; Sheu, Shuenn Jyi SIAM Journal on Control and Optimization, Vol. 25, Issue 3 https://doi.org/10.1137/0325042	journal	May 1987
Stochastic Gradient Hamiltonian Monte Carlo Chen, Tianqi; Fox, Emily B.; Guestrin, Carlos arXiv https://doi.org/10.48550/arxiv.1402.4102	preprint	January 2014
A Deterministic Gradient-Based Approach to Avoid Saddle Points Kreusser, Lisa Maria; Osher, Stanley J.; Wang, Bao arXiv https://doi.org/10.48550/arxiv.1901.06827	preprint	January 2019
The reduced-order hybrid Monte Carlo sampling smoother: The reduced-order hybrid Monte Carlo sampling smoother Attia, Ahmed; Ştefănescu, Răzvan; Sandu, Adrian International Journal for Numerical Methods in Fluids, Vol. 83, Issue 1 https://doi.org/10.1002/fld.4255	journal	June 2016
Hamiltonian Monte Carlo acceleration using surrogate functions with random bases Zhang, Cheng; Shahbaba, Babak; Zhao, Hongkai Statistics and Computing, Vol. 27, Issue 6 https://doi.org/10.1007/s11222-016-9699-1	journal	September 2016
Correlation functions and computer simulations Parisi, G. Nuclear Physics B, Vol. 180, Issue 3 https://doi.org/10.1016/0550-3213(81)90056-0	journal	May 1981
Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise Mattingly, J. C.; Stuart, A. M.; Higham, D. J. Stochastic Processes and their Applications, Vol. 101, Issue 2 https://doi.org/10.1016/S0304-4149(02)00150-3	journal	October 2002
Theoretical guarantees for approximate sampling from smooth and log-concave densities Dalalyan, Arnak S. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 79, Issue 3 https://doi.org/10.1111/rssb.12183	journal	April 2016
A Bayesian Approach to Estimating Background Flows from a Passive Scalar Borggaard, Jeff; Glatt-Holtz, Nathan; Krometis, Justin SIAM/ASA Journal on Uncertainty Quantification, Vol. 8, Issue 3 https://doi.org/10.1137/19M1267544	journal	January 2020
Nonasymptotic convergence analysis for the unadjusted Langevin algorithm Durmus, Alain; Moulines, Éric The Annals of Applied Probability, Vol. 27, Issue 3 https://doi.org/10.1214/16-AAP1238	journal	June 2017
Weighted Csiszár-Kullback-Pinsker inequalities and applications to transportation inequalities Bolley, François; Villani, Cédric Annales de la faculté des sciences de Toulouse Mathématiques, Vol. 14, Issue 3 https://doi.org/10.5802/afst.1095	journal	January 2005

Similar Records

Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Journal Article · Tue Feb 02 19:00:00 EST 2021 · Journal of Computational Physics · OSTI ID:1853726

An adaptive Hessian approximated stochastic gradient MCMC method

Journal Article · Wed Feb 03 19:00:00 EST 2021 · Journal of Computational Physics · OSTI ID:1853727

Challenges in Markov Chain Monte Carlo for Bayesian Neural Networks

Journal Article · Mon Jun 20 20:00:00 EDT 2022 · Statistical Science · OSTI ID:1976073

Related Subjects

97 MATHEMATICS AND COMPUTING
Langevin dynamics
Laplacian smoothing
stochastic gradient

Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

Citation Formats

References (12)

Similar Records

Related Subjects