Exact and inexact subsampled Newton methods for optimization
- Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL, USA
- Department of Computer Science, University of Colorado, Boulder, CO, USA
Abstract
The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how to coordinate the accuracy in the gradient and Hessian to yield a superlinear rate of convergence in expectation. The second part of the paper analyzes an inexact Newton method that solves linear systems approximately using the conjugate gradient (CG) method, and that samples the Hessian and not the gradient (the gradient is assumed to be exact). We provide a complexity analysis for this method based on the properties of the CG iteration and the quality of the Hessian approximation, and compare it with a method that employs a stochastic gradient iteration instead of the CG method. We report preliminary numerical results that illustrate the performance of inexact subsampled Newton methods on machine learning applications based on logistic regression.
- Research Organization:
- Northwestern Univ., Evanston, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- FG02-87ER25047
- OSTI ID:
- 1610078
- Journal Information:
- IMA Journal of Numerical Analysis, Vol. 39, Issue 2; ISSN 0272-4979
- Publisher:
- Oxford University Press/Institute of Mathematics and its Applications
- Country of Publication:
- United States
- Language:
- English
Similar Records
An investigation of Newton-Sketch and subsampled Newton methods
An inexact regularized Newton framework with a worst-case iteration complexity of $ {\mathscr O}(\varepsilon^{-3/2}) $ for nonconvex optimization