A Smoothed Augmented Lagrangian Framework for Convex Optimization with Nonsmooth Constraints

Zhang, Peixuan; Shanbhag, Uday V.; Fang, Ethan X.

doi:10.1007/s10915-025-02934-w

A Smoothed Augmented Lagrangian Framework for Convex Optimization with Nonsmooth Constraints

Journal Article · Sat Jun 21 00:00:00 EDT 2025 · Journal of Scientific Computing

DOI:https://doi.org/10.1007/s10915-025-02934-w· OSTI ID:3010461

Zhang, Peixuan ^[1]; Shanbhag, Uday V. ^[2]; Fang, Ethan X. ^[3]

Pennsylvania State Univ., University Park, PA (United States)
Univ. of Michigan, Ann Arbor, MI (United States)
Duke Univ., Durham, NC (United States)

Augmented Lagrangian (AL) methods have proven remarkably useful in solving optimization problems with complicated constraints. The last decade has seen the development of overall complexity guarantees for inexact AL variants. Yet, a crucial gap persists in addressing nonsmooth convex constraints. To this end, we present a smoothed augmented Lagrangian (AL) framework where nonsmooth terms are progressively smoothed with a smoothing parameter $$\eta _k$$ . The resulting AL subproblems are $$\eta _k$$ -smooth, allowing for leveraging accelerated schemes. By a careful selection of the inexactness level $$\epsilon _k$$ (for inexact subproblem resolution), the penalty parameter $$\rho _k$$ , and smoothing parameter $$\eta _k$$ at epoch k, we derive rate and complexity guarantees of $$\tilde{\mathcal {O}}(1/{\varepsilon }^{3/2})$$ and $$\tilde{\mathcal {O}}(1/{\varepsilon })$$ in convex and strongly convex regimes for computing an $${\varepsilon }$$ -optimal solution, when $$\rho _k$$ increases at a geometric rate, a significant improvement over the best available guarantees for AL schemes for convex programs with nonsmooth constraints. Analogous guarantees are developed for settings with $$\rho _k = \rho$$ as well as $$\eta _k = \eta$$ . Preliminary numerics on a fused Lasso problem display promise.

View Accepted Manuscript (DOE)

Research Organization:: Rutgers Univ., Piscataway, NJ (United States); Pennsylvania State Univ., University Park, PA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0023303

OSTI ID:: 3010461

Report Number(s):: DOE-Rutgers-23303

Journal Information:: Journal of Scientific Computing, Journal Name: Journal of Scientific Computing Journal Issue: 2 Vol. 104; ISSN 1573-7691; ISSN 0885-7474

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (31)

Multiplier and gradient methods Hestenes, Magnus R. Journal of Optimization Theory and Applications, Vol. 4, Issue 5 https://doi.org/10.1007/BF00927673	journal	November 1969
A dual approach to solving nonlinear programming problems by unconstrained optimization Rockafellar, R. Tyrrell Mathematical Programming, Vol. 5, Issue 1 https://doi.org/10.1007/BF01580138	journal	December 1973
Smooth minimization of non-smooth functions Nesterov, Yu. Mathematical Programming, Vol. 103, Issue 1 https://doi.org/10.1007/s10107-004-0552-5	journal	December 2004
Iteration-complexity of first-order augmented Lagrangian methods for convex programming Lan, Guanghui; Monteiro, Renato D. C. Mathematical Programming, Vol. 155, Issue 1-2 https://doi.org/10.1007/s10107-015-0861-x	journal	January 2015
Iteration complexity of inexact augmented Lagrangian methods for constrained convex programming Xu, Yangyang Mathematical Programming, Vol. 185, Issue 1-2 https://doi.org/10.1007/s10107-019-01425-9	journal	August 2019
Inexact accelerated augmented Lagrangian methods Kang, Myeongmin; Kang, Myungjoo; Jung, Miyoun Computational Optimization and Applications, Vol. 62, Issue 2 https://doi.org/10.1007/s10589-015-9742-8	journal	March 2015
An Efficient Algorithm for ℓ 0 Minimization in Wavelet Frame Based Image Restoration Dong, Bin; Zhang, Yong Journal of Scientific Computing, Vol. 54, Issue 2-3 https://doi.org/10.1007/s10915-012-9597-4	journal	May 2012
Adaptive inexact fast augmented Lagrangian methods for constrained convex optimization Patrascu, Andrei; Necoara, Ion; Tran-Dinh, Quoc Optimization Letters, Vol. 11, Issue 3 https://doi.org/10.1007/s11590-016-1024-6	journal	March 2016
Complexity of first-order inexact Lagrangian and penalty methods for conic convex programming Necoara, I.; Patrascu, A.; Glineur, F. Optimization Methods and Software, Vol. 34, Issue 2 https://doi.org/10.1080/10556788.2017.1380642	journal	October 2017
On the Analysis of Inexact Augmented Lagrangian Schemes for Misspecified Conic Convex Programs Aybat, Necdet Serhat; Ahmadi, Hesam; Shanbhag, Uday V. IEEE Transactions on Automatic Control, Vol. 67, Issue 8 https://doi.org/10.1109/TAC.2021.3118340	journal	August 2022
Sparsity and smoothness via the fused lasso Tibshirani, Robert; Saunders, Michael; Rosset, Saharon Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 67, Issue 1 https://doi.org/10.1111/j.1467-9868.2005.00490.x	journal	February 2005
Regularization and variable selection via the elastic net Zou, Hui; Hastie, Trevor Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 67, Issue 2 https://doi.org/10.1111/j.1467-9868.2005.00503.x	journal	April 2005
Regression Shrinkage and Selection Via the Lasso Tibshirani, Robert Journal of the Royal Statistical Society: Series B (Methodological), Vol. 58, Issue 1 https://doi.org/10.1111/j.2517-6161.1996.tb02080.x	journal	January 1996
Global and Finite Termination of a Two-Phase Augmented Lagrangian Filter Method for General Quadratic Programs Friedlander, Michael P.; Leyffer, Sven SIAM Journal on Scientific Computing, Vol. 30, Issue 4 https://doi.org/10.1137/060669930	journal	January 2008
A First-Order Smoothed Penalty Method for Compressed Sensing Aybat, N. S.; Iyengar, G. SIAM Journal on Optimization, Vol. 21, Issue 1 https://doi.org/10.1137/090762294	journal	January 2011
Multiuser Optimization: Distributed Algorithms and Error Analysis Koshal, Jayash; Nedić, Angelia; Shanbhag, Uday V. SIAM Journal on Optimization, Vol. 21, Issue 3 https://doi.org/10.1137/090770102	journal	July 2011
Smoothing and First Order Methods: A Unified Framework Beck, Amir; Teboulle, Marc SIAM Journal on Optimization, Vol. 22, Issue 2 https://doi.org/10.1137/100818327	journal	January 2012
Double Smoothing Technique for Large-Scale Linearly Constrained Convex Optimization Devolder, Olivier; Glineur, François; Nesterov, Yurii SIAM Journal on Optimization, Vol. 22, Issue 2 https://doi.org/10.1137/110826102	journal	January 2012
Computational Complexity of Inexact Gradient Augmented Lagrangian Methods: Application to Constrained MPC Nedelcu, Valentin; Necoara, Ion; Tran-Dinh, Quoc SIAM Journal on Control and Optimization, Vol. 52, Issue 5 https://doi.org/10.1137/120897547	journal	January 2014
Phase Retrieval from Incomplete Magnitude Information via Total Variation Regularization Chang, Huibin; Lou, Yifei; Ng, Michael K. SIAM Journal on Scientific Computing, Vol. 38, Issue 6 https://doi.org/10.1137/15M1029357	journal	January 2016
A Data Scalable Augmented Lagrangian KKT Preconditioner for Large-Scale Inverse Problems Alger, Nick; Villa, Umberto; Bui-Thanh, Tan SIAM Journal on Scientific Computing, Vol. 39, Issue 5 https://doi.org/10.1137/16M1084365	journal	January 2017
Parallelizable Algorithms for Optimization Problems with Orthogonality Constraints Gao, Bin; Liu, Xin; Yuan, Ya-xiang SIAM Journal on Scientific Computing, Vol. 41, Issue 3 https://doi.org/10.1137/18M1221679	journal	January 2019
Iteration-Complexity of First-Order Augmented Lagrangian Methods for Convex Conic Programming Lu, Zhaosong; Zhou, Zirui SIAM Journal on Optimization, Vol. 33, Issue 2 https://doi.org/10.1137/21M1403837	journal	June 2023
SNOPT: An SQP Algorithm for Large-Scale Constrained Optimization Gill, Philip E.; Murray, Walter; Saunders, Michael A. SIAM Review, Vol. 47, Issue 1 https://doi.org/10.1137/S0036144504446096	journal	January 2005
A Globally Convergent Linearly Constrained Lagrangian Method for Nonlinear Optimization Friedlander, Michael P.; Saunders, Michael A. SIAM Journal on Optimization, Vol. 15, Issue 3 https://doi.org/10.1137/S1052623402419789	journal	January 2005
An Interior Point Algorithm for Large-Scale Nonlinear Programming Byrd, Richard H.; Hribar, Mary E.; Nocedal, Jorge SIAM Journal on Optimization, Vol. 9, Issue 4 https://doi.org/10.1137/S1052623497325107	journal	January 1999
Solving Stochastic Optimization with Expectation Constraints Efficiently by a Stochastic Augmented Lagrangian-Type Algorithm Zhang, Liwei; Zhang, Yule; Wu, Jia INFORMS Journal on Computing, Vol. 34, Issue 6 https://doi.org/10.1287/ijoc.2022.1228	journal	November 2022
Augmented Lagrangians and Applications of the Proximal Point Algorithm in Convex Programming Rockafellar, R. T. Mathematics of Operations Research, Vol. 1, Issue 2 https://doi.org/10.1287/moor.1.2.97	journal	May 1976
On the Nonergodic Convergence Rate of an Inexact Augmented Lagrangian Framework for Composite Convex Programming Liu, Ya-Feng; Liu, Xin; Ma, Shiqian Mathematics of Operations Research, Vol. 44, Issue 2 https://doi.org/10.1287/moor.2018.0939	journal	May 2019
Smoothed Variable Sample-Size Accelerated Proximal Methods for Nonsmooth Stochastic Convex Programs Jalilzadeh, Afrooz; Shanbhag, Uday; Blanchet, Jose Stochastic Systems, Vol. 12, Issue 4 https://doi.org/10.1287/stsy.2022.0095	journal	December 2022
Proximité et dualité dans un espace hilbertien Moreau, J. J. Bulletin de la Société mathématique de France, Vol. 79 https://doi.org/10.24033/bsmf.1625	journal	January 1951

Similar Records

An adaptive sampling augmented Lagrangian method for stochastic optimization with deterministic constraints

Journal Article · Mon Oct 02 00:00:00 EDT 2023 · Computers and Mathematics with Applications (Oxford) · OSTI ID:2335998

A proximal trust-region method for nonsmooth optimization with inexact function and gradient evaluations

Journal Article · Sun Dec 25 23:00:00 EST 2022 · Mathematical Programming · OSTI ID:2311742

Related Subjects

augmented Lagrangian
convex optimization
smoothing

A Smoothed Augmented Lagrangian Framework for Convex Optimization with Nonsmooth Constraints

Citation Formats

References (31)

Similar Records

Related Subjects