A self-correcting variable-metric algorithm framework for nonsmooth optimization

Curtis, Frank E.; Robinson, Daniel P.; Zhou, Baoyu

doi:10.1093/imanum/drz008

Title: A self-correcting variable-metric algorithm framework for nonsmooth optimization

Abstract

Abstract An algorithm framework is proposed for minimizing nonsmooth functions. The framework is variable metric in that, in each iteration, a step is computed using a symmetric positive-definite matrix whose value is updated as in a quasi-Newton scheme. However, unlike previously proposed variable-metric algorithms for minimizing nonsmooth functions, the framework exploits self-correcting properties made possible through Broyden–Fletcher–Goldfarb–Shanno-type updating. In so doing, the framework does not overly restrict the manner in which the step computation matrices are updated, yet the scheme is controlled well enough that global convergence guarantees can be established. The results of numerical experiments for a few algorithms are presented to demonstrate the self-correcting behaviours that are guaranteed by the framework.

Authors:

Curtis, Frank E. ^[1]; Robinson, Daniel P. ^[2]; Zhou, Baoyu ^[1]

Department of Industrial and Systems Engineering, Lehigh University, Bethlehem, PA, USA
Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Publication Date:: 2019-03-07

Sponsoring Org.:: USDOE

OSTI Identifier:: 1498577

Grant/Contract Number:: SC0010615; DMS-1016291; DMS-1319356; IIS-1704458

Resource Type:: Published Article

Journal Name:: IMA Journal of Numerical Analysis

Additional Journal Information:: Journal Name: IMA Journal of Numerical Analysis Journal Volume: 40 Journal Issue: 2; Journal ID: ISSN 0272-4979

Publisher:: Oxford University Press

Country of Publication:: United Kingdom

Language:: English

Citation Formats


                    Curtis, Frank E., Robinson, Daniel P., and Zhou, Baoyu. A self-correcting variable-metric algorithm framework for nonsmooth optimization.  United Kingdom: N. p., 2019. 
Web.  doi:10.1093/imanum/drz008.

Copy to clipboard


                    Curtis, Frank E., Robinson, Daniel P., & Zhou, Baoyu. A self-correcting variable-metric algorithm framework for nonsmooth optimization.  United Kingdom.  https://doi.org/10.1093/imanum/drz008

Copy to clipboard


                    Curtis, Frank E., Robinson, Daniel P., and Zhou, Baoyu. Thu .  
"A self-correcting variable-metric algorithm framework for nonsmooth optimization".  United Kingdom.  https://doi.org/10.1093/imanum/drz008.

Copy to clipboard


                    
@article{osti_1498577,

  title        = {A self-correcting variable-metric algorithm framework for nonsmooth optimization},

  author       = {Curtis, Frank E. and Robinson, Daniel P. and Zhou, Baoyu},

  abstractNote = {Abstract An algorithm framework is proposed for minimizing nonsmooth functions. The framework is variable metric in that, in each iteration, a step is computed using a symmetric positive-definite matrix whose value is updated as in a quasi-Newton scheme. However, unlike previously proposed variable-metric algorithms for minimizing nonsmooth functions, the framework exploits self-correcting properties made possible through Broyden–Fletcher–Goldfarb–Shanno-type updating. In so doing, the framework does not overly restrict the manner in which the step computation matrices are updated, yet the scheme is controlled well enough that global convergence guarantees can be established. The results of numerical experiments for a few algorithms are presented to demonstrate the self-correcting behaviours that are guaranteed by the framework.},

  doi          = {10.1093/imanum/drz008},

  journal      = {IMA Journal of Numerical Analysis},

  number       = 2,

  volume       = 40,

  place        = {United Kingdom},

  year         = {2019},

  month        = {3}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Publisher's Version of Record
https://doi.org/10.1093/imanum/drz008

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Proximity control in bundle methods for convex nondifferentiable minimization
journal, January 1990

Kiwiel, Krzysztof C.
Mathematical Programming, Vol. 46, Issue 1-3
DOI: 10.1007/BF01585731

New variants of bundle methods
journal, July 1995

Lemaréchal, Claude; Nemirovskii, Arkadii; Nesterov, Yurii
Mathematical Programming, Vol. 69, Issue 1-3
DOI: 10.1007/BF01585555

Algorithms for nonlinear constraints that use lagrangian functions
journal, December 1978

Powell, M. J. D.
Mathematical Programming, Vol. 14, Issue 1
DOI: 10.1007/BF01588967

A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results
journal, February 1992

Schramm, Helga; Zowe, Jochem
SIAM Journal on Optimization, Vol. 2, Issue 1
DOI: 10.1137/0802008

The Speed of Shor's R-algorithm
journal, February 2008

Burke, J. V.; Lewis, A. S.; Overton, M. L.
IMA Journal of Numerical Analysis, Vol. 28, Issue 4
DOI: 10.1093/imanum/drn008

A family of variable-metric methods derived by variational means
journal, January 1970

Goldfarb, Donald
Mathematics of Computation, Vol. 24, Issue 109
DOI: 10.1090/S0025-5718-1970-0258249-6

Optimization of lipschitz continuous functions
journal, December 1977

Goldstein, A. A.
Mathematical Programming, Vol. 13, Issue 1
DOI: 10.1007/BF01584320

A characterization of superlinear convergence and its application to quasi-Newton methods
journal, May 1974

Dennis, J. E.; Mor{é, Jorge J.
Mathematics of Computation, Vol. 28, Issue 126
DOI: 10.1090/S0025-5718-1974-0343581-1

A Redistributed Proximal Bundle Method for Nonconvex Optimization
journal, January 2010

Hare, Warren; Sagastizábal, Claudia
SIAM Journal on Optimization, Vol. 20, Issue 5
DOI: 10.1137/090754595

A family of variable metric proximal methods
journal, January 1995

Bonnans, J. F.; Gilbert, J. Ch.; Lemaréchal, C.
Mathematical Programming, Vol. 68, Issue 1-3
DOI: 10.1007/BF01585756

Global Convergence of a Cass of Quasi-Newton Methods on Convex Problems
journal, October 1987

Byrd, Richard H.; Nocedal, Jorge; Yuan, Ya-Xiang
SIAM Journal on Numerical Analysis, Vol. 24, Issue 5
DOI: 10.1137/0724077

�ber die globale Konvergenz von Variable-Metrik-Verfahren mit nicht-exakter Schrittweitenbestimmung
journal, September 1978

Werner, Jochen
Numerische Mathematik, Vol. 31, Issue 3
DOI: 10.1007/BF01397884

A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization
journal, January 2005

Burke, James V.; Lewis, Adrian S.; Overton, Michael L.
SIAM Journal on Optimization, Vol. 15, Issue 3
DOI: 10.1137/030601296

Restricted Step and Levenberg–Marquardt Techniques in Proximal Bundle Methods for Nonconvex Nondifferentiable Optimization
journal, February 1996

Kiwiel, Krzysztof C.
SIAM Journal on Optimization, Vol. 6, Issue 1
DOI: 10.1137/0806013

An adaptive gradient sampling algorithm for non-smooth optimization
journal, September 2012

Curtis, Frank E.; Que, Xiaocun
Optimization Methods and Software, Vol. 28, Issue 6
DOI: 10.1080/10556788.2012.714781

Globally Convergent Variable Metric Method for Nonconvex Nondifferentiable Unconstrained Minimization
journal, November 2001

Vlček, J.; Lukšan, L.
Journal of Optimization Theory and Applications, Vol. 111, Issue 2
DOI: 10.1023/A:1011990503369

A bundle-Newton method for nonsmooth unconstrained minimization
journal, January 1998

Lukšan, Ladislav; Vlček, Jan
Mathematical Programming, Vol. 83, Issue 1-3
DOI: 10.1007/BF02680566

Efficiency of Proximal Bundle Methods
journal, March 2000

Kiwiel, K. C.
Journal of Optimization Theory and Applications, Vol. 104, Issue 3
DOI: 10.1023/A:1004689609425

Variable Metric Method for Minimization
journal, February 1991

Davidon, William C.
SIAM Journal on Optimization, Vol. 1, Issue 1
DOI: 10.1137/0801001

New limited memory bundle method for large-scale nonsmooth optimization
journal, December 2004

Haarala, M.; Miettinen †, K.; Mäkelä ‡, M. M.
Optimization Methods and Software, Vol. 19, Issue 6
DOI: 10.1080/10556780410001689225

Quasi-Newton Bundle-Type Methods for Nondifferentiable Convex Optimization
journal, May 1998

Mifflin, Robert; Sun, Defeng; Qi, Liqun
SIAM Journal on Optimization, Vol. 8, Issue 2
DOI: 10.1137/S1052623496303329

Globally convergent limited memory bundle method for large-scale nonsmooth optimization
journal, April 2006

Haarala, Napsu; Miettinen, Kaisa; Mäkelä, Marko M.
Mathematical Programming, Vol. 109, Issue 1
DOI: 10.1007/s10107-006-0728-2

Local and superlinear convergence of a class of variable metric methods
journal, September 1979

Ritter, K.
Computing, Vol. 23, Issue 3
DOI: 10.1007/BF02252133

Linear convergence of epsilon-subgradient descent methods for a class of convex functions
journal, September 1999

Robinson, Stephen M.
Mathematical Programming, Vol. 86, Issue 1
DOI: 10.1007/s101070050078

Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization
journal, January 2007

Kiwiel, Krzysztof C.
SIAM Journal on Optimization, Vol. 18, Issue 2
DOI: 10.1137/050639673

A Tool for the Analysis of Quasi-Newton Methods with Application to Unconstrained Minimization
journal, June 1989

Byrd, Richard H.; Nocedal, Jorge
SIAM Journal on Numerical Analysis, Vol. 26, Issue 3
DOI: 10.1137/0726042

A quasi-Newton algorithm for nonconvex, nonsmooth optimization with global convergence guarantees
journal, May 2015

Curtis, Frank E.; Que, Xiaocun
Mathematical Programming Computation, Vol. 7, Issue 4
DOI: 10.1007/s12532-015-0086-2

An Algorithm for Constrained Optimization with Semismooth Functions
journal, May 1977

Mifflin, Robert
Mathematics of Operations Research, Vol. 2, Issue 2
DOI: 10.1287/moor.2.2.191

Rate of Convergence of the Bundle Method
journal, March 2017

Du, Yu; Ruszczyński, Andrzej
Journal of Optimization Theory and Applications, Vol. 173, Issue 3
DOI: 10.1007/s10957-017-1108-1

Two-Point Step Size Gradient Methods
journal, January 1988

Barzilai, Jonathan; Borwein, Jonathan M.
IMA Journal of Numerical Analysis, Vol. 8, Issue 1
DOI: 10.1093/imanum/8.1.141

Conditioning of quasi-Newton methods for function minimization
journal, September 1970

Shanno, D. F.
Mathematics of Computation, Vol. 24, Issue 111
DOI: 10.1090/S0025-5718-1970-0274029-X

A Linearization Algorithm for Nonsmooth Minimization
journal, May 1985

Kiwiel, Krzysztof Czesław
Mathematics of Operations Research, Vol. 10, Issue 2
DOI: 10.1287/moor.10.2.185

Variable metric methods of minimisation
journal, February 1969

Pearson, J. D.
The Computer Journal, Vol. 12, Issue 2
DOI: 10.1093/comjnl/12.2.171

A Method for Solving Certain Quadratic Programming Problems Arising in Nonsmooth Optimization
journal, January 1986

Kiwiel, Krzysztof C.
IMA Journal of Numerical Analysis, Vol. 6, Issue 2
DOI: 10.1093/imanum/6.2.137

Nonsmooth optimization via quasi-Newton methods
journal, February 2012

Lewis, Adrian S.; Overton, Michael L.
Mathematical Programming, Vol. 141, Issue 1-2
DOI: 10.1007/s10107-012-0514-2

The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations
journal, January 1970

Broyden, C. G.
IMA Journal of Applied Mathematics, Vol. 6, Issue 1
DOI: 10.1093/imamat/6.1.76

A new approach to variable metric algorithms
journal, March 1970

Fletcher, R.
The Computer Journal, Vol. 13, Issue 3
DOI: 10.1093/comjnl/13.3.317

A Trust Region Spectral Bundle Method for Nonconvex Eigenvalue Optimization
journal, January 2008

Apkarian, P.; Noll, D.; Prot, O.
SIAM Journal on Optimization, Vol. 19, Issue 1
DOI: 10.1137/060665191

A -algorithm for convex minimization
journal, July 2005

Mifflin, Robert; Sagastizábal, Claudia
Mathematical Programming, Vol. 104, Issue 2-3
DOI: 10.1007/s10107-005-0630-3

Similar Records in DOE PAGES and OSTI.GOV collections:

A self-correcting variable-metric algorithm framework for nonsmooth optimization

Journal Article Curtis, Frank E. ; Robinson, Daniel P. ; Zhou, Baoyu - IMA Journal of Numerical Analysis

Abstract An algorithm framework is proposed for minimizing nonsmooth functions. The framework is variable metric in that, in each iteration, a step is computed using a symmetric positive-definite matrix whose value is updated as in a quasi-Newton scheme. However, unlike previously proposed variable-metric algorithms for minimizing nonsmooth functions, the framework exploits self-correcting properties made possible through Broyden–Fletcher–Goldfarb–Shanno-type updating. In so doing, the framework does not overly restrict the manner in which the step computation matrices are updated, yet the scheme is controlled well enough that global convergence guarantees can be established. The results of numerical experiments for a few algorithmsmore »« less
https://doi.org/10.1093/imanum/drz008
Automatic history matching with variable-metric methods

Journal Article Yang, P H ; Watson, A T - SPE (Society of Petroleum Engineers) Reservoir Engineering; (USA)

Variable-metric minimization methods are introduced for use with optimal-control theory for automatic reservoir history matching. Variable-metric methods are more efficient and robust that the previously used minimization methods, such as steepest-descent and conjugate-gradient methods. In addition, variable-metric methods can be used effectively with parameter-inequality constraints. Two variable-metric methods-the Broyden/Fletcher/Goldfarb/Shanno (BFGS) method and a self-scaling variable-metric (SSVM) method-are tested with hypothetical two-phase reservoir history-matching problems. Estimated rock/fluid properties include permeability, porosity, and relative permeabilities. The SSVM method is more efficient than the BFGS method when the number of unknown parameters is large. Both methods perform better than the steepest-descent and Nazareth'smore »« less
https://doi.org/10.2118/16977-PA
Pre-conditioned BFGS-based uncertainty quantification in elastic full-waveform inversion

Journal Article Liu, Qiancheng ; Beller, Stephen ; Lei, Wenjie ; ... - Geophysical Journal International

SUMMARY Full-waveform inversion has become an essential technique for mapping geophysical subsurface structures. However, proper uncertainty quantification is often lacking in current applications. In theory, uncertainty quantification is related to the inverse Hessian (or the posterior covariance matrix). Even for common geophysical inverse problems its calculation is beyond the computational and storage capacities of the largest high-performance computing systems. In this study, we amend the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm to perform uncertainty quantification for large-scale applications. For seismic inverse problems, the limited-memory BFGS (L-BFGS) method prevails as the most efficient quasi-Newton method. We aim to augment it further to obtain anmore »« less
https://doi.org/10.1093/gji/ggab375
Image registration with auto-mapped control volumes

Journal Article Schreibmann, Eduard ; Lei, Xing - Medical Physics

Many image registration algorithms rely on the use of homologous control points on the two input image sets to be registered. In reality, the interactive identification of the control points on both images is tedious, difficult, and often a source of error. We propose a two-step algorithm to automatically identify homologous regions that are used as a priori information during the image registration procedure. First, a number of small control volumes having distinct anatomical features are identified on the model image in a somewhat arbitrary fashion. Instead of attempting to find their correspondences in the reference image through user interaction,more »« less
https://doi.org/10.1118/1.2184440
A STRUCTURED QUASI-NEWTON ALGORITHM FOR OPTIMIZING WITH INCOMPLETE HESSIAN INFORMATION

Journal Article Petra, Cosmin G. ; Chiang, Nai-Yuan ; Anitescu, Mihai - SIAM Journal on Optimization

We present a structured quasi-Newton algorithm for unconstrained optimization problems that have unavailable second-order derivatives or Hessian terms. We provide a formal derivation of the well-known Broyden-Fletcher-Goldfarb-Shanno (BFGS) secant update formula that approximates only the missing Hessian terms, and we propose a linesearch quasi-Newton algorithm based on a modification of Wolfe conditions that converges to first-order optimality conditions. We also analyze the local convergence properties of the structured BFGS algorithm and show that it achieves superlinear convergence under the standard assumptions used by quasi-Newton methods using secant updates. We provide a thorough study of the practical performance of the algorithmmore »« less
https://doi.org/10.1137/18M1167942

Similar Records

Title: A self-correcting variable-metric algorithm framework for nonsmooth optimization

Abstract

Citation Formats

Proximity control in bundle methods for convex nondifferentiable minimization journal, January 1990

New variants of bundle methods journal, July 1995

Algorithms for nonlinear constraints that use lagrangian functions journal, December 1978

A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results journal, February 1992

The Speed of Shor's R-algorithm journal, February 2008

A family of variable-metric methods derived by variational means journal, January 1970

Optimization of lipschitz continuous functions journal, December 1977

A characterization of superlinear convergence and its application to quasi-Newton methods journal, May 1974

A Redistributed Proximal Bundle Method for Nonconvex Optimization journal, January 2010

A family of variable metric proximal methods journal, January 1995

Global Convergence of a Cass of Quasi-Newton Methods on Convex Problems journal, October 1987

�ber die globale Konvergenz von Variable-Metrik-Verfahren mit nicht-exakter Schrittweitenbestimmung journal, September 1978

A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization journal, January 2005

Restricted Step and Levenberg–Marquardt Techniques in Proximal Bundle Methods for Nonconvex Nondifferentiable Optimization journal, February 1996

An adaptive gradient sampling algorithm for non-smooth optimization journal, September 2012

Globally Convergent Variable Metric Method for Nonconvex Nondifferentiable Unconstrained Minimization journal, November 2001

A bundle-Newton method for nonsmooth unconstrained minimization journal, January 1998

Efficiency of Proximal Bundle Methods journal, March 2000

Variable Metric Method for Minimization journal, February 1991

New limited memory bundle method for large-scale nonsmooth optimization journal, December 2004

Quasi-Newton Bundle-Type Methods for Nondifferentiable Convex Optimization journal, May 1998

Globally convergent limited memory bundle method for large-scale nonsmooth optimization journal, April 2006

Local and superlinear convergence of a class of variable metric methods journal, September 1979

Linear convergence of epsilon-subgradient descent methods for a class of convex functions journal, September 1999

Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization journal, January 2007

A Tool for the Analysis of Quasi-Newton Methods with Application to Unconstrained Minimization journal, June 1989

A quasi-Newton algorithm for nonconvex, nonsmooth optimization with global convergence guarantees journal, May 2015

An Algorithm for Constrained Optimization with Semismooth Functions journal, May 1977

Rate of Convergence of the Bundle Method journal, March 2017

Two-Point Step Size Gradient Methods journal, January 1988

Conditioning of quasi-Newton methods for function minimization journal, September 1970

A Linearization Algorithm for Nonsmooth Minimization journal, May 1985

Variable metric methods of minimisation journal, February 1969

A Method for Solving Certain Quadratic Programming Problems Arising in Nonsmooth Optimization journal, January 1986

Nonsmooth optimization via quasi-Newton methods journal, February 2012

The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations journal, January 1970

A new approach to variable metric algorithms journal, March 1970

A Trust Region Spectral Bundle Method for Nonconvex Eigenvalue Optimization journal, January 2008

A -algorithm for convex minimization journal, July 2005

Proximity control in bundle methods for convex nondifferentiable minimization
journal, January 1990

New variants of bundle methods
journal, July 1995

Algorithms for nonlinear constraints that use lagrangian functions
journal, December 1978

A Version of the Bundle Idea for Minimizing a Nonsmooth Function: Conceptual Idea, Convergence Analysis, Numerical Results
journal, February 1992

The Speed of Shor's R-algorithm
journal, February 2008

A family of variable-metric methods derived by variational means
journal, January 1970

Optimization of lipschitz continuous functions
journal, December 1977

A characterization of superlinear convergence and its application to quasi-Newton methods
journal, May 1974

A Redistributed Proximal Bundle Method for Nonconvex Optimization
journal, January 2010

A family of variable metric proximal methods
journal, January 1995

Global Convergence of a Cass of Quasi-Newton Methods on Convex Problems
journal, October 1987

�ber die globale Konvergenz von Variable-Metrik-Verfahren mit nicht-exakter Schrittweitenbestimmung
journal, September 1978

A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization
journal, January 2005

Restricted Step and Levenberg–Marquardt Techniques in Proximal Bundle Methods for Nonconvex Nondifferentiable Optimization
journal, February 1996

An adaptive gradient sampling algorithm for non-smooth optimization
journal, September 2012

Globally Convergent Variable Metric Method for Nonconvex Nondifferentiable Unconstrained Minimization
journal, November 2001

A bundle-Newton method for nonsmooth unconstrained minimization
journal, January 1998

Efficiency of Proximal Bundle Methods
journal, March 2000

Variable Metric Method for Minimization
journal, February 1991

New limited memory bundle method for large-scale nonsmooth optimization
journal, December 2004

Quasi-Newton Bundle-Type Methods for Nondifferentiable Convex Optimization
journal, May 1998

Globally convergent limited memory bundle method for large-scale nonsmooth optimization
journal, April 2006

Local and superlinear convergence of a class of variable metric methods
journal, September 1979

Linear convergence of epsilon-subgradient descent methods for a class of convex functions
journal, September 1999

Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization
journal, January 2007

A Tool for the Analysis of Quasi-Newton Methods with Application to Unconstrained Minimization
journal, June 1989

A quasi-Newton algorithm for nonconvex, nonsmooth optimization with global convergence guarantees
journal, May 2015

An Algorithm for Constrained Optimization with Semismooth Functions
journal, May 1977

Rate of Convergence of the Bundle Method
journal, March 2017

Two-Point Step Size Gradient Methods
journal, January 1988

Conditioning of quasi-Newton methods for function minimization
journal, September 1970

A Linearization Algorithm for Nonsmooth Minimization
journal, May 1985

Variable metric methods of minimisation
journal, February 1969

A Method for Solving Certain Quadratic Programming Problems Arising in Nonsmooth Optimization
journal, January 1986

Nonsmooth optimization via quasi-Newton methods
journal, February 2012

The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations
journal, January 1970

A new approach to variable metric algorithms
journal, March 1970

A Trust Region Spectral Bundle Method for Nonconvex Eigenvalue Optimization
journal, January 2008

A -algorithm for convex minimization
journal, July 2005