An investigation of Newton-Sketch and subsampled Newton methods

Berahas, Albert S.; Bollapragada, Raghu; Nocedal, Jorge

doi:10.1080/10556788.2020.1725751

An investigation of Newton-Sketch and subsampled Newton methods

Journal Article · Wed Feb 12 00:00:00 EST 2020 · Optimization Methods and Software

DOI:https://doi.org/10.1080/10556788.2020.1725751· OSTI ID:1657509

^[1]; ^[2]; ^[3]

Lehigh Univ., Bethlehem, PA (United States)
Argonne National Lab. (ANL), Lemont, IL (United States)
Northwestern Univ., Evanston, IL (United States)

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. In this work, we study two forms of sketching that perform dimensionality reduction in data space: Hessian subsampling and randomized Hadamard transformations. Each has its own advantages, and their relative tradeoffs have not been investigated in the optimization literature. Additionally, our study focuses on practical versions of the two methods in which the resulting linear systems of equations are solved approximately, at every iteration, using an iterative solver. The advantages of using the conjugate gradient method vs. a stochastic gradient iteration are revealed through a set of numerical experiments, and a complexity analysis of the Hessian subsampling method is presented.

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: National Science Foundation (NSF); US Office of Naval Research (ONR); USDOD Defense Advanced Research Projects Agency (DARPA); USDOE

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1657509

Journal Information:: Optimization Methods and Software, Journal Name: Optimization Methods and Software Journal Issue: 4 Vol. 35; ISSN 1055-6788

Publisher:: Taylor & FrancisCopyright Statement

Country of Publication:: United States

Language:: English

References (14)

Second-Order Stochastic Optimization for Machine Learning in Linear Time Agarwal, Naman; Bullins, Brian; Hazan, Elad arXiv https://doi.org/10.48550/arxiv.1602.03943	text	January 2016
Numerical Optimization Nocedal, Jorge; Wright, Stephen J. Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/978-0-387-40065-5	book	January 2006
Numerical Optimization Nocedal, Jorge; Wright, Stephen J. Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/b98874	book	January 1999
Faster least squares approximation Drineas, Petros; Mahoney, Michael W.; Muthukrishnan, S. Numerische Mathematik, Vol. 117, Issue 2 https://doi.org/10.1007/s00211-010-0331-6	journal	October 2010
Coordinate descent algorithms Wright, Stephen J. Mathematical Programming, Vol. 151, Issue 1 https://doi.org/10.1007/s10107-015-0892-3	journal	March 2015
Sub-sampled Newton methods Roosta-Khorasani, Farbod; Mahoney, Michael W. Mathematical Programming, Vol. 174, Issue 1-2 https://doi.org/10.1007/s10107-018-1346-5	journal	November 2018
Exact and inexact subsampled Newton methods for optimization Bollapragada, Raghu; Byrd, Richard H.; Nocedal, Jorge IMA Journal of Numerical Analysis, Vol. 39, Issue 2 https://doi.org/10.1093/imanum/dry009	journal	April 2018
New and Improved Johnson–Lindenstrauss Embeddings via the Restricted Isometry Property Krahmer, Felix; Ward, Rachel SIAM Journal on Mathematical Analysis, Vol. 43, Issue 3 https://doi.org/10.1137/100810447	journal	January 2011
Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic Convergence Pilanci, Mert; Wainwright, Martin J. SIAM Journal on Optimization, Vol. 27, Issue 1 https://doi.org/10.1137/15M1021106	journal	January 2017
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
Sketching meets random projection in the dual: A provable recovery algorithm for big and high-dimensional data Wang, Jialei; Lee, Jason D.; Mahdavi, Mehrdad Electronic Journal of Statistics, Vol. 11, Issue 2 https://doi.org/10.1214/17-EJS1334SI	journal	January 2017
A Stochastic Approximation Method Robbins, Herbert; Monro, Sutton The Annals of Mathematical Statistics, Vol. 22, Issue 3 https://doi.org/10.1214/aoms/1177729586	journal	September 1951
Computational Advertising: Techniques for Targeting Relevant Ads Woodruff, David P. Foundations and Trends® in Theoretical Computer Science, Vol. 10, Issue 1-2 https://doi.org/10.1561/0400000060	journal	January 2014
Exact and Inexact Subsampled Newton Methods for Optimization Bollapragada, Raghu; Byrd, Richard; Nocedal, Jorge arXiv https://doi.org/10.48550/arxiv.1609.08502	preprint	January 2016

Cited By (14)

Newton-MR: Inexact Newton Method with minimum residual sub-problem solver Roosta, Fred; Liu, Yang; Xu, Peng EURO Journal on Computational Optimization, Vol. 10 https://doi.org/10.1016/j.ejco.2022.100035	journal	January 2022
Regularization by denoising sub-sampled Newton method for spectral CT multi-material decomposition Perelli, Alessandro; Andersen, Martin S. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 379, Issue 2200 https://doi.org/10.1098/rsta.2020.0191	journal	May 2021
Accelerating Adaptive Cubic Regularization of Newton's Method via Random Sampling Chen, Xi; Jiang, Bo; Lin, Tianyi arXiv https://doi.org/10.48550/arxiv.1802.05426	preprint	January 2018
Adaptive Cubic Regularization Methods with Dynamic Inexact Hessian Information and Applications to Finite-Sum Minimization Bellavia, Stefania; Gurioli, Gianmarco; Morini, Benedetta arXiv https://doi.org/10.48550/arxiv.1808.06239	preprint	January 2018
Convergence Analysis of Inexact Randomized Iterative Methods Loizou, Nicolas; Richtárik, Peter arXiv https://doi.org/10.48550/arxiv.1903.07971	preprint	January 2019
OverSketched Newton: Fast Convex Optimization for Serverless Systems Gupta, Vipul; Kadhe, Swanand; Courtade, Thomas arXiv https://doi.org/10.48550/arxiv.1903.08857	preprint	January 2019
A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization Yang, Minghan; Milzarek, Andre; Wen, Zaiwen arXiv https://doi.org/10.48550/arxiv.1910.09373	preprint	January 2019
M-IHS: An Accelerated Randomized Preconditioning Method Avoiding Costly Matrix Decompositions Ozaslan, Ibrahim Kurban; Pilanci, Mert; Arikan, Orhan arXiv https://doi.org/10.48550/arxiv.1912.03514	preprint	January 2019
A Multilevel Approach to Training Braglia, Vanessa; Kopaničáková, Alena; Krause, Rolf arXiv https://doi.org/10.48550/arxiv.2006.15602	preprint	January 2020
Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems Cartis, Coralia; Ferguson, Tyler; Roberts, Lindon arXiv https://doi.org/10.48550/arxiv.2007.13243	text	January 2020
Generalization of Quasi-Newton Methods: Application to Robust Symmetric Multisecant Updates Scieur, Damien; Liu, Lewis; Pumir, Thomas arXiv https://doi.org/10.48550/arxiv.2011.03358	preprint	January 2020
An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangians Na, Sen; Anitescu, Mihai; Kolar, Mladen arXiv https://doi.org/10.48550/arxiv.2102.05320	preprint	January 2021
Scalable Subspace Methods for Derivative-Free Nonlinear Least-Squares Optimization Cartis, Coralia; Roberts, Lindon arXiv https://doi.org/10.48550/arxiv.2102.12016	preprint	January 2021
A Multilevel Method for Self-Concordant Minimization Tsipinakis, Nick; Parpas, Panos arXiv https://doi.org/10.48550/arxiv.2106.13690	preprint	January 2021

Similar Records

Exact and inexact subsampled Newton methods for optimization

Journal Article · Tue Apr 03 00:00:00 EDT 2018 · IMA Journal of Numerical Analysis · OSTI ID:1610078

An inexact semismooth Newton method with application to adaptive randomized sketching for dynamic optimization

Journal Article · Tue Oct 17 20:00:00 EDT 2023 · Finite Elements in Analysis and Design · OSTI ID:2311364

A Scalable Interior‐Point Gauss–Newton Method for PDE‐Constrained Optimization With Bound Constraints

Journal Article · Sat Nov 29 19:00:00 EST 2025 · Numerical Linear Algebra with Applications · OSTI ID:3014007

Related Subjects

97 MATHEMATICS AND COMPUTING
Newton's method
machine learning
sketching
stochastic optimization
subsampling

An investigation of Newton-Sketch and subsampled Newton methods

Citation Formats

References (14)

Cited By (14)

Similar Records

Related Subjects