On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces

Vuchkov, Radoslav G.; Petra, Cosmin G.; Petra, Noémi

doi:10.1080/01630563.2020.1785496

Title: On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces

Abstract

Newton’s method is usually preferred when solving optimization problems due to its superior convergence properties compared to gradient-based or derivative-free optimization algorithms. However, deriving and computing second-order derivatives needed by Newton’s method often is not trivial and, in some cases, not possible. In such cases quasi-Newton algorithms are a great alternative. In this paper, we provide a new derivation of well-known quasi-Newton formulas in an infinite-dimensional Hilbert space setting. Furthermore, it is known that quasi-Newton update formulas are solutions to certain variational problems over the space of symmetric matrices. In this paper, we formulate similar variational problems over the space of bounded symmetric operators in Hilbert spaces. By changing the constraints of the variational problem we obtain updates (for the Hessian and Hessian inverse) not only for the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method but also for Davidon–Fletcher–Powell (DFP), Symmetric Rank One (SR1), and Powell-Symmetric-Broyden (PSB). In addition, for an inverse problem governed by a partial differential equation (PDE), we derive DFP and BFGS “structured” secant formulas that explicitly use the derivative of the regularization and only approximates the second derivative of the misfit term. We show numerical results that demonstrate the desired mesh-independence property and superior performance of the resulting quasi-Newtonmore » methods.« less

Authors:

Vuchkov, Radoslav G. ^[1]; Petra, Cosmin G. ^[2];

^[1]

Univ. of California, Merced, CA (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States). Center for Applied Scientific Computing

Publication Date:: Mon Jul 13 00:00:00 EDT 2020

Research Org.:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)

OSTI Identifier:: 1804288

Report Number(s):: LLNL-JRNL-818069
Journal ID: ISSN 0163-0563; 1027937

Grant/Contract Number:: AC52-07NA27344; 1654311

Resource Type:: Accepted Manuscript

Journal Name:: Numerical Functional Analysis and Optimization

Additional Journal Information:: Journal Volume: 41; Journal Issue: 13; Journal ID: ISSN 0163-0563

Publisher:: Taylor & Francis

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; BFGS; DFP; optimization in infinite dimensions; PDE-constrained optimization; PSB; quasi-Newton; SR1; variational problems

Citation Formats


                    Vuchkov, Radoslav G., Petra, Cosmin G., and Petra, Noémi. On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces.  United States: N. p., 2020. 
Web.  doi:10.1080/01630563.2020.1785496.

Copy to clipboard


                    Vuchkov, Radoslav G., Petra, Cosmin G., & Petra, Noémi. On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces.  United States.  https://doi.org/10.1080/01630563.2020.1785496

Copy to clipboard


                    Vuchkov, Radoslav G., Petra, Cosmin G., and Petra, Noémi. Mon .  
"On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces".  United States.  https://doi.org/10.1080/01630563.2020.1785496.  https://www.osti.gov/servlets/purl/1804288.

Copy to clipboard


                    
@article{osti_1804288,

  title        = {On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces},

  author       = {Vuchkov, Radoslav G. and Petra, Cosmin G. and Petra, Noémi},

  abstractNote = {Newton’s method is usually preferred when solving optimization problems due to its superior convergence properties compared to gradient-based or derivative-free optimization algorithms. However, deriving and computing second-order derivatives needed by Newton’s method often is not trivial and, in some cases, not possible. In such cases quasi-Newton algorithms are a great alternative. In this paper, we provide a new derivation of well-known quasi-Newton formulas in an infinite-dimensional Hilbert space setting. Furthermore, it is known that quasi-Newton update formulas are solutions to certain variational problems over the space of symmetric matrices. In this paper, we formulate similar variational problems over the space of bounded symmetric operators in Hilbert spaces. By changing the constraints of the variational problem we obtain updates (for the Hessian and Hessian inverse) not only for the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method but also for Davidon–Fletcher–Powell (DFP), Symmetric Rank One (SR1), and Powell-Symmetric-Broyden (PSB). In addition, for an inverse problem governed by a partial differential equation (PDE), we derive DFP and BFGS “structured” secant formulas that explicitly use the derivative of the regularization and only approximates the second derivative of the misfit term. We show numerical results that demonstrate the desired mesh-independence property and superior performance of the resulting quasi-Newton methods.},

  doi          = {10.1080/01630563.2020.1785496},

  journal      = {Numerical Functional Analysis and Optimization},

  number       = 13,

  volume       = 41,

  place        = {United States},

  year         = {Mon Jul 13 00:00:00 EDT 2020},

  month        = {Mon Jul 13 00:00:00 EDT 2020}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1080/01630563.2020.1785496

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

A family of variable metric methods in function space, without exact line searches
journal, July 1980

Mayorga, R. V.; Quintana, V. H.
Journal of Optimization Theory and Applications, Vol. 31, Issue 3
DOI: 10.1007/BF01262975

Representations of quasi-Newton matrices and their use in limited memory methods
journal, January 1994

Byrd, Richard H.; Nocedal, Jorge; Schnabel, Robert B.
Mathematical Programming, Vol. 63, Issue 1-3
DOI: 10.1007/BF01582063

Davidon’s Method in Hilbert Space
journal, July 1968

Horwitz, Lawrence B.; Sarachik, Philip E.
SIAM Journal on Applied Mathematics, Vol. 16, Issue 4
DOI: 10.1137/0116055

Duality in quasi-Newton methods and new variational characterizations of the DFP and BFGS updates
journal, February 2009

Güler, Osman; Gürtuna, Filiz; Shevchenko, Olena
Optimization Methods and Software, Vol. 24, Issue 1
DOI: 10.1080/10556780802367205

Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence
journal, January 2005

Wächter, Andreas; Biegler, Lorenz T.
SIAM Journal on Optimization, Vol. 16, Issue 1
DOI: 10.1137/S1052623403426556

Line Search Filter Methods for Nonlinear Programming: Local Convergence
journal, January 2005

Wächter, Andreas; Biegler, Lorenz T.
SIAM Journal on Optimization, Vol. 16, Issue 1
DOI: 10.1137/S1052623403426544

Two-Point Step Size Gradient Methods
journal, January 1988

Barzilai, Jonathan; Borwein, Jonathan M.
IMA Journal of Numerical Analysis, Vol. 8, Issue 1
DOI: 10.1093/imanum/8.1.141

From Functional Analysis to Iterative Methods
journal, January 2010

Kirby, Robert C.
SIAM Review, Vol. 52, Issue 2
DOI: 10.1137/070706914

A generalization of the Sherman–Morrison–Woodbury formula
journal, September 2011

Deng, Chun Yuan
Applied Mathematics Letters, Vol. 24, Issue 9
DOI: 10.1016/j.aml.2011.03.046

Applied Analysis
journal, February 2001

Hunter, John K.; Nachtergaele, Bruno
DOI: 10.1142/4319

Quasi-Newton methods in infinite-dimensional spaces and application to matrix equations
journal, June 2010

Benahmed, Boubakeur; Mokhtar-Kharroubi, Hocine; de Malafosse, Bruno
Journal of Global Optimization, Vol. 49, Issue 3
DOI: 10.1007/s10898-010-9564-2

Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations
journal, January 2011

Flath, H. P.; Wilcox, L. C.; Akçelik, V.
SIAM Journal on Scientific Computing, Vol. 33, Issue 1
DOI: 10.1137/090780717

Computational Methods for Inverse Problems
book, January 2002

Vogel, Curtis R.
Frontiers in Applied Mathematics
DOI: 10.1137/1.9780898717570

Broyden's method in Hilbert space
journal, May 1986

Sachs, Ekkehard W.
Mathematical Programming, Vol. 35, Issue 1
DOI: 10.1007/BF01589442

Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves
journal, April 2012

Bui-Thanh, Tan; Ghattas, Omar
Inverse Problems, Vol. 28, Issue 5
DOI: 10.1088/0266-5611/28/5/055002

Quasi-Newton Methods, Motivation and Theory
journal, January 1977

Dennis, Jr., J. E.; Moré, Jorge J.
SIAM Review, Vol. 19, Issue 1
DOI: 10.1137/1019005

On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming
journal, April 2005

Wächter, Andreas; Biegler, Lorenz T.
Mathematical Programming, Vol. 106, Issue 1
DOI: 10.1007/s10107-004-0559-y

Similar Records in DOE PAGES and OSTI.GOV collections:

A STRUCTURED QUASI-NEWTON ALGORITHM FOR OPTIMIZING WITH INCOMPLETE HESSIAN INFORMATION

Journal Article Petra, Cosmin G. ; Chiang, Nai-Yuan ; Anitescu, Mihai - SIAM Journal on Optimization

We present a structured quasi-Newton algorithm for unconstrained optimization problems that have unavailable second-order derivatives or Hessian terms. We provide a formal derivation of the well-known Broyden-Fletcher-Goldfarb-Shanno (BFGS) secant update formula that approximates only the missing Hessian terms, and we propose a linesearch quasi-Newton algorithm based on a modification of Wolfe conditions that converges to first-order optimality conditions. We also analyze the local convergence properties of the structured BFGS algorithm and show that it achieves superlinear convergence under the standard assumptions used by quasi-Newton methods using secant updates. We provide a thorough study of the practical performance of the algorithmmore »« less
https://doi.org/10.1137/18M1167942
A Structured Quasi-Newton Algorithm for Optimizing with Incomplete Hessian Information

Journal Article Petra, Cosmin G. ; Chiang, Naiyuan ; Anitescu, Mihai - SIAM Journal on Optimization

We present a structured quasi-Newton algorithm for unconstrained optimization problems that have unavailable second-order derivatives or Hessian terms. We provide a formal derivation of the well-known Broyden--Fletcher--Goldfarb--Shanno (BFGS) secant update formula that approximates only the missing Hessian terms, and we propose a linesearch quasi-Newton algorithm based on a modification of Wolfe conditions that converges to first-order optimality conditions. We also analyze the local convergence properties of the structured BFGS algorithm and show that it achieves superlinear convergence under the standard assumptions used by quasi-Newton methods using secant updates. In conclusion, we provide a thorough study of the practical performance ofmore »« less
Cited by 5
https://doi.org/10.1137/18M1167942

Full Text Available
Extension of quasi-Newton methods to constrained optimization and to general systems of nonlinear equations and inequalities. Final report, February 1, 1976--January 31, 1977

Technical Report Tapia, R A

A theory for extending the popular quasi-Newton secant methods, including the Davidon--Fletcher--Powell method and the Broyden--Fletcher--Goldfarb--Shanno method, for unconstrained minimization to constrained minimization was developed. Since these standard methods must work with positive definite matrices and the usual formulation of the first-order conditions for constrained optimization problems does not lead to a positive definite matrix, a very important part of the theory was the exploitation of positive definite subproblems. The theory is modular, and as a consequence one can now extend an algorithm for unconstrained optimization to constrained optimization in an effective manner.
Pre-conditioned BFGS-based uncertainty quantification in elastic full-waveform inversion

Journal Article Liu, Qiancheng ; Beller, Stephen ; Lei, Wenjie ; ... - Geophysical Journal International

SUMMARY Full-waveform inversion has become an essential technique for mapping geophysical subsurface structures. However, proper uncertainty quantification is often lacking in current applications. In theory, uncertainty quantification is related to the inverse Hessian (or the posterior covariance matrix). Even for common geophysical inverse problems its calculation is beyond the computational and storage capacities of the largest high-performance computing systems. In this study, we amend the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm to perform uncertainty quantification for large-scale applications. For seismic inverse problems, the limited-memory BFGS (L-BFGS) method prevails as the most efficient quasi-Newton method. We aim to augment it further to obtain anmore »« less
https://doi.org/10.1093/gji/ggab375
Compact representations of structured BFGS matrices

Journal Article Brust, Johannes J. ; Di, Zichao ; Leyffer, Sven ; ... - Computational Optimization and Applications

For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, recent structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. This article develops the compact representations of two structured Broyden-Fletcher-Goldfarb-Shanno update formulas. The compact representations enable efficient limited memory and initialization strategies. Two limited memory line search algorithms are described for which extensive numerical results demonstrate the efficacy of the algorithms, including comparisons to IPOPT on large machine learning problems, and to L-BFGS on a real worldmore »« less
https://doi.org/10.1007/s10589-021-00297-0

Full Text Available

Similar Records

Title: On the Derivation of Quasi-Newton Formulas for Optimization in Function Spaces

Abstract

Citation Formats

A family of variable metric methods in function space, without exact line searches journal, July 1980

Representations of quasi-Newton matrices and their use in limited memory methods journal, January 1994

Davidon’s Method in Hilbert Space journal, July 1968

Duality in quasi-Newton methods and new variational characterizations of the DFP and BFGS updates journal, February 2009

Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence journal, January 2005

Line Search Filter Methods for Nonlinear Programming: Local Convergence journal, January 2005

Two-Point Step Size Gradient Methods journal, January 1988

From Functional Analysis to Iterative Methods journal, January 2010

A generalization of the Sherman–Morrison–Woodbury formula journal, September 2011

Applied Analysis journal, February 2001

Quasi-Newton methods in infinite-dimensional spaces and application to matrix equations journal, June 2010

Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations journal, January 2011

Computational Methods for Inverse Problems book, January 2002

Broyden's method in Hilbert space journal, May 1986

Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves journal, April 2012

Quasi-Newton Methods, Motivation and Theory journal, January 1977

On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming journal, April 2005

A family of variable metric methods in function space, without exact line searches
journal, July 1980

Representations of quasi-Newton matrices and their use in limited memory methods
journal, January 1994

Davidon’s Method in Hilbert Space
journal, July 1968

Duality in quasi-Newton methods and new variational characterizations of the DFP and BFGS updates
journal, February 2009

Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence
journal, January 2005

Line Search Filter Methods for Nonlinear Programming: Local Convergence
journal, January 2005

Two-Point Step Size Gradient Methods
journal, January 1988

From Functional Analysis to Iterative Methods
journal, January 2010

A generalization of the Sherman–Morrison–Woodbury formula
journal, September 2011

Applied Analysis
journal, February 2001

Quasi-Newton methods in infinite-dimensional spaces and application to matrix equations
journal, June 2010

Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations
journal, January 2011

Computational Methods for Inverse Problems
book, January 2002

Broyden's method in Hilbert space
journal, May 1986

Analysis of the Hessian for inverse scattering problems: II. Inverse medium scattering of acoustic waves
journal, April 2012

Quasi-Newton Methods, Motivation and Theory
journal, January 1977

On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming
journal, April 2005