Robust and Simple ADMM Penalty Parameter Selection

McCann, Michael Thompson; Wohlberg, Brendt Egon

doi:10.1109/ojsp.2023.3349115

Robust and Simple ADMM Penalty Parameter Selection

Journal Article · Wed Jan 10 04:00:00 EST 2024 · IEEE Open Journal of Signal Processing (Online)

DOI:https://doi.org/10.1109/ojsp.2023.3349115· OSTI ID:2283368

^[1]; ^[1]

Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

We present a new method for online selection of the penalty parameter for the alternating direction method of multipliers (ADMM) algorithm. ADMM is a widely used method for solving a range of optimization problems, including those that arise in signal and image processing. In its standard form, ADMM includes a scalar hyperparameter, known as the penalty parameter, which usually has to be tuned to achieve satisfactory empirical convergence. In this work, we develop a framework for analyzing the ADMM algorithm applied to a quadratic problem as an affine fixed point iteration. Using this framework, we develop a new method for automatically tuning the penalty parameter by detecting when it has become too large or small. We analyze this and several other methods with respect to their theoretical properties, i.e., robustness to problem transformations, and empirical performance on several optimization problems. Our proposed algorithm is based on a theoretical framework with clear, explicit assumptions and approximations, is theoretically covariant/invariant to problem transformations, is simple to implement, and exhibits competitive empirical performance.

View Accepted Manuscript (DOE)

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001

OSTI ID:: 2283368

Report Number(s):: LA-UR--23-29141

Journal Information:: IEEE Open Journal of Signal Processing (Online), Journal Name: IEEE Open Journal of Signal Processing (Online) Vol. 5; ISSN 2644-1322

Publisher:: Institute of Electrical and Electronics Engineers Inc. (IEEE)Copyright Statement

Country of Publication:: United States

Language:: English

References (30)

Introductory Lectures on Convex Optimization Nesterov, Yurii Applied Optimization https://doi.org/10.1007/978-1-4419-8853-9	book	January 2004
Proximal Splitting Methods in Signal Processing Combettes, Patrick L.; Pesquet, Jean-Christophe Springer Optimization and Its Applications https://doi.org/10.1007/978-1-4419-9569-8_10	book	January 2011
Self Equivalence of the Alternating Direction Method of Multipliers Yan, Ming; Yin, Wotao Splitting Methods in Communication, Imaging, Science, and Engineering https://doi.org/10.1007/978-3-319-41589-5_5	book	January 2016
Non-stationary Douglas–Rachford and alternating direction method of multipliers: adaptive step-sizes and convergence Lorenz, Dirk A.; Tran-Dinh, Quoc Computational Optimization and Applications, Vol. 74, Issue 1 https://doi.org/10.1007/s10589-019-00106-9	journal	May 2019
On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers Deng, Wei; Yin, Wotao Journal of Scientific Computing, Vol. 66, Issue 3 https://doi.org/10.1007/s10915-015-0048-x	journal	May 2015
Global Convergence of ADMM in Nonconvex Nonsmooth Optimization Wang, Yu; Yin, Wotao; Zeng, Jinshan Journal of Scientific Computing, Vol. 78, Issue 1 https://doi.org/10.1007/s10915-018-0757-z	journal	June 2018
A dual algorithm for the solution of nonlinear variational problems via finite element approximation Gabay, Daniel; Mercier, Bertrand Computers & Mathematics with Applications, Vol. 2, Issue 1 https://doi.org/10.1016/0898-1221(76)90003-1	journal	January 1976
Nuclear norm system identification with missing inputs and outputs Liu, Zhang; Hansson, Anders; Vandenberghe, Lieven Systems & Control Letters, Vol. 62, Issue 8 https://doi.org/10.1016/j.sysconle.2013.04.005	journal	August 2013
Linear Analysis Bollobás, Béla https://doi.org/10.1017/CBO9781139168472	book	January 1999
Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities He, B. S.; Yang, H.; Wang, S. L. Journal of Optimization Theory and Applications, Vol. 106, Issue 2, p. 337-356 https://doi.org/10.1023/A:1004603514434	journal	August 2000
Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité d'une classe de problèmes de Dirichlet non linéaires Glowinski, R.; Marroco, A. Revue française d'automatique, informatique, recherche opérationnelle. Analyse numérique, Vol. 9, Issue R2 https://doi.org/10.1051/m2an/197509r200411	journal	January 1975
XDesign : an open-source software package for designing X-ray imaging phantoms and experiments Ching, Daniel J.; Gürsoy, Dogˇa Journal of Synchrotron Radiation, Vol. 24, Issue 2 https://doi.org/10.1107/S1600577517001928	journal	February 2017
Alternating direction method of multipliers for strictly convex quadratic programs: Optimal parameter selection Raghunathan, Arvind U.; Di Cairano, Stefano 2014 American Control Conference https://doi.org/10.1109/ACC.2014.6859093	conference	June 2014
Subspace system identification via weighted nuclear norm optimization Hansson, Anders; Liu, Zhang; Vandenberghe, Lieven 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) https://doi.org/10.1109/CDC.2012.6426980	conference	December 2012
Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation Xu, Zheng; Figueiredo, Mario A. T.; Yuan, Xiaoming 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.765	conference	July 2017
Efficient convolutional sparse coding Wohlberg, Brendt 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2014.6854992	conference	May 2014
Phase retrieval of sparse signals using optimization transfer and ADMM Weller, Daniel S.; Pnueli, Ayelet; Radzyner, Ori 2014 IEEE International Conference on Image Processing (ICIP) https://doi.org/10.1109/ICIP.2014.7025268	conference	October 2014
ADMM penalty parameter selection with krylov subspace recycling technique for sparse coding Lin, Youzuo; Wohlberg, Brendt; Vesselinov, Velimir 2017 IEEE International Conference on Image Processing (ICIP) https://doi.org/10.1109/ICIP.2017.8296621	conference	September 2017
Optimal Parameter Selection for the Alternating Direction Method of Multipliers (ADMM): Quadratic Problems Ghadimi, Euhanna; Teixeira, Andre; Shames, Iman IEEE Transactions on Automatic Control, Vol. 60, Issue 3 https://doi.org/10.1109/TAC.2014.2354892	journal	March 2015
Collaborative Sparse Regression for Hyperspectral Unmixing Iordache, Marian-Daniel; Bioucas-Dias, Jose M.; Plaza, Antonio IEEE Transactions on Geoscience and Remote Sensing, Vol. 52, Issue 1 https://doi.org/10.1109/TGRS.2013.2240001	journal	January 2014
A New Alternating Minimization Algorithm for Total Variation Image Reconstruction Wang, Yilun; Yang, Junfeng; Yin, Wotao SIAM Journal on Imaging Sciences, Vol. 1, Issue 3 https://doi.org/10.1137/080724265	journal	January 2008
The Split Bregman Method for L1-Regularized Problems Goldstein, Tom; Osher, Stanley SIAM Journal on Imaging Sciences, Vol. 2, Issue 2 https://doi.org/10.1137/080725891	journal	January 2009
A Singular Value Thresholding Algorithm for Matrix Completion Cai, Jian-Feng; Candès, Emmanuel J.; Shen, Zuowei SIAM Journal on Optimization, Vol. 20, Issue 4 https://doi.org/10.1137/080738970	journal	January 2010
Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints Lessard, Laurent; Recht, Benjamin; Packard, Andrew SIAM Journal on Optimization, Vol. 26, Issue 1 https://doi.org/10.1137/15M1009597	journal	January 2016
Atomic Decomposition by Basis Pursuit Chen, Scott Shaobing; Donoho, David L.; Saunders, Michael A. SIAM Review, Vol. 43, Issue 1 https://doi.org/10.1137/S003614450037906X	journal	January 2001
Robust principal component analysis? Candès, Emmanuel J.; Li, Xiaodong; Ma, Yi Journal of the ACM, Vol. 58, Issue 3 https://doi.org/10.1145/1970392.1970395	journal	May 2011
Alternating Direction Method of Multipliers for Separable Convex Optimization of Real Functions in Complex Variables Li, Lu; Wang, Xingyu; Wang, Guoqiang Mathematical Problems in Engineering, Vol. 2015 https://doi.org/10.1155/2015/104531	journal	January 2015
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers Boyd, Stephen; Parikh, Neal; Chu, Eric https://doi.org/10.1561/2200000016	book	January 2011
Proximal Algorithms Parikh, Neal Foundations and Trends® in Optimization, Vol. 1, Issue 3 https://doi.org/10.1561/2400000003	journal	January 2014
Matrix Computations Golub, Gene H.; Van Loan, Charles F. https://doi.org/10.56021/9781421407944	book	February 2013

Similar Records

Learning-Accelerated ADMM for Distributed DC Optimal Power Flow

Conference · Wed Jul 28 00:00:00 EDT 2021 · OSTI ID:1817363

Learning-Accelerated ADMM for Distributed DC Optimal Power Flow

Journal Article · Mon Dec 14 23:00:00 EST 2020 · IEEE Control Systems Letters · OSTI ID:1755744

Scalable Preconditioning of Block-Structured Linear Algebra Systems using ADMM

Journal Article · Tue Jun 18 00:00:00 EDT 2019 · Computers and Chemical Engineering · OSTI ID:1570258

Related Subjects

42 ENGINEERING
ADMM
adaptive ADMM
convex optimization
parameter selection
penalty parameter

Robust and Simple ADMM Penalty Parameter Selection

Citation Formats

References (30)

Similar Records

Related Subjects