Accelerated Sparse Recovery via Gradient Descent with Nonlinear Conjugate Gradient Momentum

Hu, Mengqi; Lou, Yifei; Wang, Bao; Yan, Ming; Yang, Xiu; Ye, Qiang

doi:10.1007/s10915-023-02148-y

Title: Accelerated Sparse Recovery via Gradient Descent with Nonlinear Conjugate Gradient Momentum

Journal Article · 08 March 2023 · Journal of Scientific Computing

DOI: https://doi.org/10.1007/s10915-023-02148-y · OSTI ID:2280650

Hu, Mengqi ^[1];

^[2]; Wang, Bao ^[3]; Yan, Ming ^[4]; Yang, Xiu ^[1]; Ye, Qiang ^[5]

Lehigh University, Bethlehem, PA (United States)
University of Texas at Dallas, Richardson, TX (United States)
University of Utah, Salt Lake City, UT (United States)
Chinese University of Hong Kong, Guangdong (China); Michigan State University, East Lansing, MI (United States)
University of Kentucky, Lexington, KY (United States)

This paper applies an idea of adaptive momentum for the nonlinear conjugate gradient to accelerate optimization problems in sparse recovery. Specifically, we consider two types of minimization problems: a (single) differentiable function and the sum of a non-smooth function and a differentiable function. In the first case, we adopt a fixed step size to avoid the traditional line search and establish the convergence analysis of the proposed algorithm for a quadratic problem. This acceleration is further incorporated with an operator splitting technique to deal with the non-smooth function in the second case. As a result, we use the convex ι₁ and the nonconvex ι₁ – ι₂ functionals as two case studies to demonstrate the efficiency of the proposed approaches over traditional methods.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Univ. of Utah, Salt Lake City, UT (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF)

Grant/Contract Number:: SC0021142; SC0002722; DMS-1821144; DMS-1924935; DMS-1952339; DMS-2152762; DMS-2208361; DMS-2012439

OSTI ID:: 2280650

Journal Information:: Journal of Scientific Computing, Vol. 95, Issue 1; ISSN 0885-7474

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (54)

Signal Recovery by Proximal Forward-Backward Splitting Combettes, Patrick L.; Wajs, Valérie R. Multiscale Modeling & Simulation, Vol. 4, Issue 4 https://doi.org/10.1137/050626090	journal	January 2005
A fast iterative thresholding algorithm for wavelet-regularized deconvolution Vonesch, Cédric; Unser, Michael SPIE Proceedings https://doi.org/10.1117/12.733532	conference	September 2007
Sparse Reconstruction by Separable Approximation Wright, S. J.; Nowak, R. D.; Figueiredo, M. A. T. IEEE Transactions on Signal Processing, Vol. 57, Issue 7 https://doi.org/10.1109/TSP.2009.2016892	journal	July 2009
Computing Sparse Representation in a Highly Coherent Dictionary Based on Difference of $$L_1$$ L 1 and $$L_2$$ L 2 Lou, Yifei; Yin, Penghang; He, Qi Journal of Scientific Computing, Vol. 64, Issue 1 https://doi.org/10.1007/s10915-014-9930-1	journal	October 2014
Another hybrid conjugate gradient algorithm for unconstrained optimization Andrei, Neculai Numerical Algorithms, Vol. 47, Issue 2 https://doi.org/10.1007/s11075-007-9152-9	journal	January 2008
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint Daubechies, I.; Defrise, M.; De Mol, C. Communications on Pure and Applied Mathematics, Vol. 57, Issue 11 https://doi.org/10.1002/cpa.20042	journal	January 2004
Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage Chambolle, A.; De Vore, R. A. IEEE Transactions on Image Processing, Vol. 7, Issue 3 https://doi.org/10.1109/83.661182	journal	March 1998
Inexact Preconditioned Conjugate Gradient Method with Inner-Outer Iteration Golub, Gene H.; Ye, Qiang SIAM Journal on Scientific Computing, Vol. 21, Issue 4 https://doi.org/10.1137/S1064827597323415	journal	January 1999
Compressed sensing Donoho, D. L. IEEE Transactions on Information Theory, Vol. 52, Issue 4 https://doi.org/10.1109/TIT.2006.871582	journal	April 2006
Fast L1–L2 Minimization via a Proximal Operator Lou, Yifei; Yan, Ming Journal of Scientific Computing, Vol. 74, Issue 2 https://doi.org/10.1007/s10915-017-0463-2	journal	May 2017
Iterative Methods for Sparse Linear Systems Saad, Yousef https://doi.org/10.1137/1.9780898718003	book	January 2003
An EM algorithm for wavelet-based image restoration Figueiredo, M. A. T.; Nowak, R. D. IEEE Transactions on Image Processing, Vol. 12, Issue 8 https://doi.org/10.1109/TIP.2003.814255	journal	August 2003
Sparse Approximate Solutions to Linear Systems Natarajan, B. K. SIAM Journal on Computing, Vol. 24, Issue 2 https://doi.org/10.1137/S0097539792240406	journal	April 1995
A Scale-Invariant Approach for Sparse Signal Recovery Rahimi, Yaghoub; Wang, Chao; Dong, Hongbo SIAM Journal on Scientific Computing, Vol. 41, Issue 6 https://doi.org/10.1137/18M123147X	journal	January 2019
IDENT: Identifying Differential Equations with Numerical Time Evolution Kang, Sung Ha; Liao, Wenjing; Liu, Yingjie Journal of Scientific Computing, Vol. 87, Issue 1 https://doi.org/10.1007/s10915-020-01404-9	journal	February 2021
Point Source Super-resolution Via Non-convex $$L_1$$ Based Methods Lou, Yifei; Yin, Penghang; Xin, Jack Journal of Scientific Computing, Vol. 68, Issue 3 https://doi.org/10.1007/s10915-016-0169-x	journal	January 2016
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems Beck, Amir; Teboulle, Marc SIAM Journal on Imaging Sciences, Vol. 2, Issue 1 https://doi.org/10.1137/080716542	journal	January 2009
Restart procedures for the conjugate gradient method Powell, M. J. D. Mathematical Programming, Vol. 12, Issue 1 https://doi.org/10.1007/BF01593790	journal	December 1977
Accelerated Schemes for the $L_1/L_2$ Minimization Wang, Chao; Yan, Ming; Rahimi, Yaghoub IEEE Transactions on Signal Processing, Vol. 68 https://doi.org/10.1109/TSP.2020.2985298	journal	January 2020
Ratio and difference of $l_1$ and $l_2$ norms and sparse representation with coherent dictionaries Yin, Penghang; Esser, Ernie; Xin, Jack Communications in Information and Systems, Vol. 14, Issue 2 https://doi.org/10.4310/CIS.2014.v14.n2.a2	journal	January 2014
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers Boyd, Stephen; Parikh, Neal; Chu, Eric https://doi.org/10.1561/2200000016	book	January 2011
Fitting Data with Errors in All Variables Using the Huber M-estimator Hermey, Damaris; Watson, G. Alistair SIAM Journal on Scientific Computing, Vol. 20, Issue 4 https://doi.org/10.1137/S106482759731823X	journal	January 1999
Methods of conjugate gradients for solving linear systems Hestenes, M. R.; Stiefel, E. Journal of Research of the National Bureau of Standards, Vol. 49, Issue 6 https://doi.org/10.6028/jres.049.044	journal	December 1952
Nonconvex Sorted $$\ell _1$$ ℓ 1 Minimization for Sparse Approximation Huang, Xiao-Lin; Shi, Lei; Yan, Ming Journal of the Operations Research Society of China, Vol. 3, Issue 2 https://doi.org/10.1007/s40305-014-0069-4	journal	February 2015
Function minimization by conjugate gradients Fletcher, R. The Computer Journal, Vol. 7, Issue 2 https://doi.org/10.1093/comjnl/7.2.149	journal	February 1964
Descent Property and Global Convergence of the Fletcher—Reeves Method with Inexact Line Search Al-Baali, M. IMA Journal of Numerical Analysis, Vol. 5, Issue 1 https://doi.org/10.1093/imanum/5.1.121	journal	January 1985
Global Convergence Properties of Conjugate Gradient Methods for Optimization Gilbert, Jean Charles; Nocedal, Jorge SIAM Journal on Optimization, Vol. 2, Issue 1 https://doi.org/10.1137/0802003	journal	February 1992
Minimization of $\ell_{1-2}$ for Compressed Sensing Yin, Penghang; Lou, Yifei; He, Qi SIAM Journal on Scientific Computing, Vol. 37, Issue 1 https://doi.org/10.1137/140952363	journal	January 2015
A smoothing conjugate gradient method for solving systems of nonsmooth equations Narushima, Yasushi Applied Mathematics and Computation, Vol. 219, Issue 16 https://doi.org/10.1016/j.amc.2013.02.060	journal	April 2013
Some methods of speeding up the convergence of iteration methods Polyak, B. T. USSR Computational Mathematics and Mathematical Physics, Vol. 4, Issue 5 https://doi.org/10.1016/0041-5553(64)90137-5	journal	January 1964
Likelihood-Based Selection and Sharp Parameter Estimation Shen, Xiaotong; Pan, Wei; Zhu, Yunzhang Journal of the American Statistical Association, Vol. 107, Issue 497 https://doi.org/10.1080/01621459.2011.645783	journal	March 2012
The smoothing Fletcher-Reeves conjugate gradient method for solving finite minimax problems Pang, Deyan; Du, Shouqiang; Ju, Jingjie ScienceAsia, Vol. 42, Issue 1 https://doi.org/10.2306/scienceasia1513-1874.2016.42.040	journal	January 2016
Sharpness, Restart, and Acceleration Roulet, Vincent; d'Aspremont, Alexandre SIAM Journal on Optimization, Vol. 30, Issue 1 https://doi.org/10.1137/18M1224568	journal	January 2020
Proximal Algorithms Parikh, Neal Foundations and Trends® in Optimization, Vol. 1, Issue 3 https://doi.org/10.1561/2400000003	journal	January 2014
A Generalized Krylov Subspace Method for $\ell_p$-$\ell_q$ Minimization Lanza, A.; Morigi, S.; Reichel, L. SIAM Journal on Scientific Computing, Vol. 37, Issue 5 https://doi.org/10.1137/140967982	journal	January 2015
Stochastic Collocation Methods via Minimisation of the Transformed L₁-Penalty Guo, Ling East Asian Journal on Applied Mathematics, Vol. 8, Issue 3 https://doi.org/10.4208/eajam.060518.130618	journal	June 2018
Monotonicity and restart in fast gradient methods Giselsson, Pontus; Boyd, Stephen 53rd IEEE Conference on Decision and Control https://doi.org/10.1109/CDC.2014.7040179	conference	December 2014
A D.C. Optimization Algorithm for Solving the Trust-Region Subproblem Tao, Pham Dinh; An, Le Thi Hoai SIAM Journal on Optimization, Vol. 8, Issue 2 https://doi.org/10.1137/S1052623494274313	journal	May 1998
Iterative reweighted minimization methods for $$l_p$$ l p regularized unconstrained nonlinear programming Lu, Zhaosong Mathematical Programming, Vol. 147, Issue 1-2 https://doi.org/10.1007/s10107-013-0722-4	journal	November 2013
Minimization of transformed $$L_1$$ L 1 penalty: theory, difference of convex function algorithm, and robust application in compressed sensing Zhang, Shuai; Xin, Jack Mathematical Programming, Vol. 169, Issue 1 https://doi.org/10.1007/s10107-018-1236-x	journal	March 2018
Half-Quadratic Algorithm for $ℓ_p-ℓ_q$ Problems with Applications to TV-$ℓ_1$ Image Restoration and Compressive Sensing Chan, Raymond H.; Liang, Hai-Xia Efficient Algorithms for Global Optimization Methods in Computer Vision https://doi.org/10.1007/978-3-642-54774-4_4	book	January 2014
Minimization of transformed $l_1$ penalty: Closed form representation and iterative thresholding algorithms Zhang, Shuai; Xin, Jack Communications in Mathematical Sciences, Vol. 15, Issue 2 https://doi.org/10.4310/CMS.2017.v15.n2.a9	journal	January 2017
Majorization–minimization generalized Krylov subspace methods for $${\ell _p}$$–$${\ell _q}$$ optimization applied to image restoration Huang, G.; Lanza, A.; Morigi, S. BIT Numerical Mathematics, Vol. 57, Issue 2 https://doi.org/10.1007/s10543-016-0643-8	journal	January 2017
A new class of nonlinear conjugate gradient coefficients with exact and inexact line searches Rivaie, Mohd; Mamat, Mustafa; Abashar, Abdelrhaman Applied Mathematics and Computation, Vol. 268 https://doi.org/10.1016/j.amc.2015.07.019	journal	October 2015
A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search Hager, William W.; Zhang, Hongchao SIAM Journal on Optimization, Vol. 16, Issue 1 https://doi.org/10.1137/030601880	journal	January 2005
A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property Dai, Y. H.; Yuan, Y. SIAM Journal on Optimization, Vol. 10, Issue 1 https://doi.org/10.1137/S1052623497318992	journal	January 1999
The place of the in robust estimation Huber, Peter J. Computational Statistics & Data Analysis, Vol. 5, Issue 4 https://doi.org/10.1016/0167-9473(87)90049-1	journal	September 1987
Constructing Test Instances for Basis Pursuit Denoising Lorenz, D. A. IEEE Transactions on Signal Processing, Vol. 61, Issue 5 https://doi.org/10.1109/TSP.2012.2236322	journal	March 2013
Sampling-50 years after Shannon Unser, M. Proceedings of the IEEE, Vol. 88, Issue 4 https://doi.org/10.1109/5.843002	journal	April 2000
Enhancing Sparsity by Reweighted ℓ 1 Minimization Candès, Emmanuel J.; Wakin, Michael B.; Boyd, Stephen P. Journal of Fourier Analysis and Applications, Vol. 14, Issue 5-6 https://doi.org/10.1007/s00041-008-9045-x	journal	October 2008
Adaptive Huber Regression Sun, Qiang; Zhou, Wen-Xin; Fan, Jianqing Journal of the American Statistical Association, Vol. 115, Issue 529 https://doi.org/10.1080/01621459.2018.1543124	journal	April 2019
Analysis of the finite precision bi-conjugate gradient algorithm for nonsymmetric linear systems Tong, Charles H.; Ye, Qiang Mathematics of Computation, Vol. 69, Issue 232 https://doi.org/10.1090/S0025-5718-99-01171-0	journal	August 1999
Minimization of functions having Lipschitz continuous first partial derivatives Armijo, Larry Pacific Journal of Mathematics, Vol. 16, Issue 1 https://doi.org/10.2140/pjm.1966.16.1	journal	January 1966
Smoothing Nonlinear Conjugate Gradient Method for Image Restoration Using Nonsmooth Nonconvex Minimization Chen, Xiaojun; Zhou, Weijun SIAM Journal on Imaging Sciences, Vol. 3, Issue 4 https://doi.org/10.1137/080740167	journal	January 2010

Similar Records

Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum

Journal Article · 2023 · IEEE Transactions on Neural Networks and Learning Systems · OSTI ID:2280651

Wang, Bao; Ye, Qiang

Bayesian sparse learning with preconditioned stochastic gradient MCMC and its applications

Journal Article · 2021 · Journal of Computational Physics · OSTI ID:1853726

Wang, Yating; Deng, Wei; Lin, Guang

Glassoformer: A Query-Sparse Transformer for Post-Fault Power Grid Voltage Prediction

Journal Article · 2022 · Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing · OSTI ID:1866753

Zheng, Yunling; Hu, Carson; Lin, Guang; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Accelerated gradient momentum
Operator splitting
Fixed step size
Convergence rate

Title: Accelerated Sparse Recovery via Gradient Descent with Nonlinear Conjugate Gradient Momentum

Citation Formats

References (54)

Similar Records

Related Subjects