Inexact Newton-CG algorithms with complexity guarantees

Yao, Zhewei; Xu, Peng; Roosta, Fred; Wright, Stephen J.; Mahoney, Michael W.

doi:10.1093/imanum/drac043

Inexact Newton-CG algorithms with complexity guarantees

Journal Article · Mon Aug 22 00:00:00 EDT 2022 · IMA Journal of Numerical Analysis

DOI:https://doi.org/10.1093/imanum/drac043· OSTI ID:2424937

Yao, Zhewei; Xu, Peng; Roosta, Fred; Wright, Stephen J.; Mahoney, Michael W.

Abstract

We consider variants of a recently developed Newton-CG algorithm for nonconvex problems (Royer, C. W. & Wright, S. J. (2018) Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim., 28, 1448–1477) in which inexact estimates of the gradient and the Hessian information are used for various steps. Under certain conditions on the inexactness measures, we derive iteration complexity bounds for achieving $$\epsilon $$-approximate second-order optimality that match best-known lower bounds. Our inexactness condition on the gradient is adaptive, allowing for crude accuracy in regions with large gradients. We describe two variants of our approach, one in which the step size along the computed search direction is chosen adaptively, and another in which the step size is pre-defined. To obtain second-order optimality, our algorithms will make use of a negative curvature direction on some steps. These directions can be obtained, with high probability, using the randomized Lanczos algorithm. In this sense, all of our results hold with high probability over the run of the algorithm. We evaluate the performance of our proposed algorithms empirically on several machine learning models. Our approach is a first attempt to introduce inexact Hessian and/or gradient information into the Newton-CG algorithm of Royer & Wright (2018, Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim., 28, 1448–1477).

Research Organization:: US Department of Energy (USDOE), Washington, DC (United States). Office of Science, Exascale Computing Project

Sponsoring Organization:: USDOE

OSTI ID:: 2424937

Journal Information:: IMA Journal of Numerical Analysis, Journal Name: IMA Journal of Numerical Analysis Journal Issue: 3 Vol. 43; ISSN 0272-4979

Publisher:: Oxford University Press/Institute of Mathematics and its Applications

Country of Publication:: United States

Language:: English

References (23)

Adaptive cubic regularisation methods for unconstrained optimization. Part I: motivation, convergence and numerical results Cartis, Coralia; Gould, Nicholas I. M.; Toint, Philippe L. Mathematical Programming, Vol. 127, Issue 2 https://doi.org/10.1007/s10107-009-0286-5	journal	May 2009
Sub-sampled Newton methods Roosta-Khorasani, Farbod; Mahoney, Michael W. Mathematical Programming, Vol. 174, Issue 1-2 https://doi.org/10.1007/s10107-018-1346-5	journal	November 2018
A Stochastic Line Search Method with Expected Complexity Analysis Paquette, Courtney; Scheinberg, Katya SIAM Journal on Optimization, Vol. 30, Issue 1 https://doi.org/10.1137/18M1216250	journal	January 2020
Efficient BackProp LeCun, Yann A.; Bottou, Léon; Orr, Genevieve B. Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-35289-8_3	book	January 2012
Cubic regularization of Newton method and its global performance Nesterov, Yurii; Polyak, B. T. Mathematical Programming, Vol. 108, Issue 1 https://doi.org/10.1007/s10107-006-0706-8	journal	April 2006
A Newton-CG algorithm with complexity guarantees for smooth unconstrained optimization Royer, Clément W.; O’Neill, Michael; Wright, Stephen J. Mathematical Programming, Vol. 180, Issue 1-2 https://doi.org/10.1007/s10107-019-01362-7	journal	January 2019
Newton-type methods for non-convex optimization under inexact Hessian information Xu, Peng; Roosta, Fred; Mahoney, Michael W. Mathematical Programming, Vol. 184, Issue 1-2 https://doi.org/10.1007/s10107-019-01405-z	journal	May 2019
Some NP-complete problems in quadratic and nonlinear programming Murty, Katta G.; Kabadi, Santosh N. Mathematical Programming, Vol. 39, Issue 2 https://doi.org/10.1007/BF02592948	journal	June 1987
Adaptive Regularization Algorithms with Inexact Evaluations for Nonconvex Optimization Bellavia, Stefania; Gurioli, Gianmarco; Morini, Benedetta SIAM Journal on Optimization, Vol. 29, Issue 4 https://doi.org/10.1137/18M1226282	journal	January 2019
Most Tensor Problems Are NP-Hard Hillar, Christopher J.; Lim, Lek-Heng Journal of the ACM, Vol. 60, Issue 6 https://doi.org/10.1145/2512329	journal	November 2013
Complexity and global rates of trust-region methods based on probabilistic models Gratton, Serge; Royer, Clément W.; Vicente, Luís N. IMA Journal of Numerical Analysis, Vol. 38, Issue 3 https://doi.org/10.1093/imanum/drx043	journal	August 2017
Adaptive cubic regularisation methods for unconstrained optimization. Part II: worst-case function- and derivative-evaluation complexity Cartis, Coralia; Gould, Nicholas I. M.; Toint, Philippe L. Mathematical Programming, Vol. 130, Issue 2 https://doi.org/10.1007/s10107-009-0337-y	journal	January 2010
Convergence of Newton-MR under Inexact Hessian Information Liu, Yang; Roosta, Fred SIAM Journal on Optimization, Vol. 31, Issue 1 https://doi.org/10.1137/19M1302211	journal	January 2021
The Conjugate Gradient Method and Trust Regions in Large Scale Optimization Steihaug, Trond SIAM Journal on Numerical Analysis, Vol. 20, Issue 3 https://doi.org/10.1137/0720042	journal	June 1983
LIBSVM: A library for support vector machines Chang, Chih-Chung; Lin, Chih-Jen ACM Transactions on Intelligent Systems and Technology, Vol. 2, Issue 3 https://doi.org/10.1145/1961189.1961199	journal	April 2011
Global convergence rate analysis of unconstrained optimization methods based on probabilistic models Cartis, C.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1137-4	journal	April 2017
Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales Blanchet, Jose; Cartis, Coralia; Menickelly, Matt INFORMS Journal on Optimization, Vol. 1, Issue 2 https://doi.org/10.1287/ijoo.2019.0016	journal	April 2019
Trust-Region Newton-CG with Strong Second-Order Complexity Guarantees for Nonconvex Optimization Curtis, Frank E.; Robinson, Daniel P.; Royer, Clément W. SIAM Journal on Optimization, Vol. 31, Issue 1 https://doi.org/10.1137/19M130563X	journal	January 2021
Complexity Analysis of Second-Order Line-Search Algorithms for Smooth Nonconvex Optimization Royer, Clément W.; Wright, Stephen J. SIAM Journal on Optimization, Vol. 28, Issue 2 https://doi.org/10.1137/17M1134329	journal	January 2018
Second-order Optimization for Non-convex Machine Learning: an Empirical Study Xu, Peng; Roosta, Fred; Mahoney, Michael W. Proceedings of the 2020 SIAM International Conference on Data Mining https://doi.org/10.1137/1.9781611976236.23	book	January 2020
Complexity bounds for second-order optimality in unconstrained optimization Cartis, C.; Gould, N. I. M.; Toint, Ph. L. Journal of Complexity, Vol. 28, Issue 1 https://doi.org/10.1016/j.jco.2011.06.001	journal	February 2012
First-order and Stochastic Optimization Methods for Machine Learning Lan, Guanghui Springer Series in the Data Sciences https://doi.org/10.1007/978-3-030-39568-1	book	January 2020
Trust Region Methods Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe L. MOS-SIAM Series on Optimization https://doi.org/10.1137/1.9780898719857	book	January 2000

Similar Records

Exact and inexact subsampled Newton methods for optimization

Journal Article · Tue Apr 03 00:00:00 EDT 2018 · IMA Journal of Numerical Analysis · OSTI ID:1610078

An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

Journal Article · Tue May 08 00:00:00 EDT 2018 · IMA Journal of Numerical Analysis · OSTI ID:1611471

An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization

Journal Article · Tue May 08 00:00:00 EDT 2018 · IMA Journal of Numerical Analysis · OSTI ID:1436375

Related Subjects

Mathematics

Inexact Newton-CG algorithms with complexity guarantees

Citation Formats

References (23)

Similar Records

Related Subjects