Compact representations of structured BFGS matrices

Brust, Johannes J.; Di, Zichao; Leyffer, Sven; Petra, Cosmin G.

doi:10.1007/s10589-021-00297-0

Compact representations of structured BFGS matrices

Journal Article · Fri Jul 09 00:00:00 EDT 2021 · Computational Optimization and Applications

DOI:https://doi.org/10.1007/s10589-021-00297-0· OSTI ID:1824771

^[1]; Di, Zichao ^[1]; Leyffer, Sven ^[1]; Petra, Cosmin G. ^[2]

Argonne National Lab. (ANL), Argonne, IL (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, recent structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. Here, this article develops the compact representations of two structured Broyden-Fletcher-Goldfarb-Shanno update formulas. The compact representations enable efficient limited memory and initialization strategies. Two limited memory line search algorithms are described for which extensive numerical results demonstrate the efficacy of the algorithms, including comparisons to IPOPT on large machine learning problems, and to L-BFGS on a real world large scale ptychographic imaging application.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States); Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357; AC52-07NA27344

OSTI ID:: 1824771

Alternate ID(s):: OSTI ID: 1863253

Report Number(s):: LLNL-JRNL--819824; 1031027

Journal Information:: Computational Optimization and Applications, Journal Name: Computational Optimization and Applications Journal Issue: 1 Vol. 80; ISSN 0926-6003

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (25)

Convergence theory for the structured BFGS secant method with an application to nonlinear least squares Dennis, J. E.; Martinez, H. J.; Tapia, R. A. Journal of Optimization Theory and Applications, Vol. 61, Issue 2 https://doi.org/10.1007/BF00962795	journal	May 1989
Representations of quasi-Newton matrices and their use in limited memory methods Byrd, Richard H.; Nocedal, Jorge; Schnabel, Robert B. Mathematical Programming, Vol. 63, Issue 1-3 https://doi.org/10.1007/BF01582063	journal	January 1994
Factorized quasi-Newton methods for nonlinear least squares problems Yabe, Hiroshi; Takahashi, Toshihiko Mathematical Programming, Vol. 51, Issue 1-3 https://doi.org/10.1007/BF01586927	journal	July 1991
Numerical Optimization Nocedal, Jorge; Wright, Stephen J. Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/b98874	book	January 1999
On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming Wächter, Andreas; Biegler, Lorenz T. Mathematical Programming, Vol. 106, Issue 1 https://doi.org/10.1007/s10107-004-0559-y	journal	April 2005
Benchmarking optimization software with performance profiles Dolan, Elizabeth D.; Moré, Jorge J. Mathematical Programming, Vol. 91, Issue 2 https://doi.org/10.1007/s101070100263	journal	January 2002
A dense initialization for limited-memory quasi-Newton methods Brust, Johannes; Burdakov, Oleg; Erway, Jennifer B. Computational Optimization and Applications, Vol. 74, Issue 1 https://doi.org/10.1007/s10589-019-00112-x	journal	May 2019
A complex gradient operator and its application in adaptive array theory Brandwood, D. H. IEE Proceedings F Communications, Radar and Signal Processing, Vol. 130, Issue 1 https://doi.org/10.1049/ip-f-1.1983.0003	journal	January 1983
A family of variable-metric methods derived by variational means Goldfarb, Donald Mathematics of Computation, Vol. 24, Issue 109 https://doi.org/10.1090/S0025-5718-1970-0258249-6	journal	January 1970
Conditioning of quasi-Newton methods for function minimization Shanno, D. F. Mathematics of Computation, Vol. 24, Issue 111 https://doi.org/10.1090/S0025-5718-1970-0274029-X	journal	September 1970
Updating quasi-Newton matrices with limited storage Nocedal, Jorge Mathematics of Computation, Vol. 35, Issue 151 https://doi.org/10.1090/S0025-5718-1980-0572855-7	journal	September 1980
A new approach to variable metric algorithms Fletcher, R. The Computer Journal, Vol. 13, Issue 3 https://doi.org/10.1093/comjnl/13.3.317	journal	March 1970
The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations Broyden, C. G. IMA Journal of Applied Mathematics, Vol. 6, Issue 1 https://doi.org/10.1093/imamat/6.1.76	journal	January 1970
Two-Point Step Size Gradient Methods Barzilai, Jonathan; Borwein, Jonathan M. IMA Journal of Numerical Analysis, Vol. 8, Issue 1 https://doi.org/10.1093/imanum/8.1.141	journal	January 1988
On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning Byrd, Richard H.; Chin, Gillian M.; Neveitt, Will SIAM Journal on Optimization, Vol. 21, Issue 3 https://doi.org/10.1137/10079923X	journal	July 2011
Quasi-Newton Methods, Motivation and Theory Dennis, Jr., J. E.; Moré, Jorge J. SIAM Review, Vol. 19, Issue 1 https://doi.org/10.1137/1019005	journal	January 1977
Unconstrained Optimization of Real Functions in Complex Variables Sorber, Laurent; Barel, Marc Van; Lathauwer, Lieven De SIAM Journal on Optimization, Vol. 22, Issue 3 https://doi.org/10.1137/110832124	journal	January 2012
Adaptive, Limited-Memory BFGS Algorithms for Unconstrained Optimization Boggs, Paul T.; Byrd, Richard H. SIAM Journal on Optimization, Vol. 29, Issue 2 https://doi.org/10.1137/16M1065100	journal	January 2019
A Structured Quasi-Newton Algorithm for Optimizing with Incomplete Hessian Information Petra, Cosmin G.; Chiang, Naiyuan; Anitescu, Mihai SIAM Journal on Optimization, Vol. 29, Issue 2 https://doi.org/10.1137/18M1167942	journal	January 2019
Multigrid Optimization for Large-Scale Ptychographic Phase Retrieval Fung, Samy Wu; Wendy, Zichao SIAM Journal on Imaging Sciences, Vol. 13, Issue 1 https://doi.org/10.1137/18M1223915	journal	January 2020
Line search algorithms with guaranteed sufficient decrease Moré, Jorge J.; Thuente, David J. ACM Transactions on Mathematical Software, Vol. 20, Issue 3 https://doi.org/10.1145/192115.192132	journal	September 1994
LIBSVM: A library for support vector machines Chang, Chih-Chung; Lin, Chih-Jen ACM Transactions on Intelligent Systems and Technology, Vol. 2, Issue 3 https://doi.org/10.1145/1961189.1961199	journal	April 2011
Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization Zhu, Ciyou; Byrd, Richard H.; Lu, Peihuang ACM Transactions on Mathematical Software, Vol. 23, Issue 4 https://doi.org/10.1145/279232.279236	journal	December 1997
An Adaptive Nonlinear Least-Squares Algorithm Dennis, John E.; Gay, David M.; Walsh, Roy E. ACM Transactions on Mathematical Software, Vol. 7, Issue 3 https://doi.org/10.1145/355958.355965	journal	September 1981
CUTEr and SifDec: A constrained and unconstrained testing environment, revisited Gould, Nicholas I. M.; Orban, Dominique; Toint, Philippe L. ACM Transactions on Mathematical Software, Vol. 29, Issue 4 https://doi.org/10.1145/962437.962439	journal	December 2003

Similar Records

A STRUCTURED QUASI-NEWTON ALGORITHM FOR OPTIMIZING WITH INCOMPLETE HESSIAN INFORMATION

Journal Article · Mon Dec 31 23:00:00 EST 2018 · SIAM Journal on Optimization · OSTI ID:1573037

A Structured Quasi-Newton Algorithm for Optimizing with Incomplete Hessian Information

Journal Article · Wed Apr 10 20:00:00 EDT 2019 · SIAM Journal on Optimization · OSTI ID:1574637

Related Subjects

97 MATHEMATICS AND COMPUTING
BFGS method
Compact representation
Large-scale optimization
Limited memory method
Quasi-Newton method
compact representation
large-scale optimization
limited memory method

Compact representations of structured BFGS matrices

Citation Formats

References (25)

Similar Records

Related Subjects