Manifold Sampling for Optimizing Nonsmooth Nonconvex Compositions

Larson, Jeffrey; Menickelly, Matt; Zhou, Baoyu

doi:10.1137/20m1378089

Manifold Sampling for Optimizing Nonsmooth Nonconvex Compositions

Journal Article · Thu Oct 28 00:00:00 EDT 2021 · SIAM Journal on Optimization

DOI:https://doi.org/10.1137/20m1378089· OSTI ID:1839939

^[1]; ^[1]; Zhou, Baoyu ^[2]

Argonne National Lab. (ANL), Argonne, IL (United States)
Lehigh Univ., Bethlehem, PA (United States)

Here we propose a manifold sampling algorithm for minimizing a nonsmooth composition $$f= h\circ F$$, where we assume $$h$$ is nonsmooth and may be inexpensively computed in closed form and $$F$$ is smooth but its Jacobian may not be available. We additionally assume that the composition $$h\circ F$$ defines a continuous selection. Manifold sampling algorithms can be classified as model-based derivative-free methods, in that models of $$F$$ are combined with particularly sampled information about $$h$$ to yield local models for use within a trust-region framework. We demonstrate that cluster points of the sequence of iterates generated by the manifold sampling algorithm are Clarke stationary. We consider the tractability of three particular subproblems generated by the manifold sampling algorithm and the extent to which inexact solutions to these subproblems may be tolerated. Numerical results demonstrate that manifold sampling as a derivative-free algorithm is competitive with state-of-the-art algorithms for nonsmooth optimization that utilize first-order information about $$f$$.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR). Scientific Discovery through Advanced Computing (SciDAC)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1839939

Journal Information:: SIAM Journal on Optimization, Journal Name: SIAM Journal on Optimization Journal Issue: 4 Vol. 31; ISSN 1052-6234

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (46)

An algorithm for composite nonsmooth optimization problems Womersley, R. S.; Fletcher, R. Journal of Optimization Theory and Applications, Vol. 48, Issue 3 https://doi.org/10.1007/bf00940574	journal	March 1986
Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization Bagirov, A. M.; Karasözen, B.; Sezer, M. Journal of Optimization Theory and Applications, Vol. 137, Issue 2 https://doi.org/10.1007/s10957-007-9335-5	journal	December 2007
Derivative-free optimization methods Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. Acta Numerica, Vol. 28 https://doi.org/10.1017/s0962492919000060	journal	May 2019
First- and second-order optimality conditions for piecewise smooth objective functions Griewank, A.; Walther, A. Optimization Methods and Software, Vol. 31, Issue 5 https://doi.org/10.1080/10556788.2016.1189549	journal	April 2016
A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles Curtis, Frank E.; Mitchell, Tim; Overton, Michael L. Optimization Methods and Software, Vol. 32, Issue 1 https://doi.org/10.1080/10556788.2016.1208749	journal	July 2016
Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 18, Issue 2 https://doi.org/10.1137/050639673	journal	January 2007
Restricted Step and Levenberg–Marquardt Techniques in Proximal Bundle Methods for Nonconvex Nondifferentiable Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 6, Issue 1 https://doi.org/10.1137/0806013	journal	February 1996
Characterizing and Testing Subdifferential Regularity in Piecewise Smooth Optimization Walther, Andrea; Griewank, Andreas SIAM Journal on Optimization, Vol. 29, Issue 2 https://doi.org/10.1137/17m115520x	journal	January 2019
An algorithm for composite nonsmooth optimization problems Womersley, R. S.; Fletcher, R. Journal of Optimization Theory and Applications, Vol. 48, Issue 3 https://doi.org/10.1007/BF00940574	journal	March 1986
A trust region algorithm for minimization of locally Lipschitzian functions Qi, Liqun; Sun, Jie Mathematical Programming, Vol. 66, Issue 1-3 https://doi.org/10.1007/BF01581136	journal	August 1994
Conditions for convergence of trust region algorithms for nonsmooth optimization Yuan, Y. Mathematical Programming, Vol. 31, Issue 2 https://doi.org/10.1007/BF02591750	journal	June 1985
On the superlinear convergence of a trust region algorithm for nonsmooth optimization Yuan, Y. Mathematical Programming, Vol. 31, Issue 3 https://doi.org/10.1007/BF02591949	journal	October 1985
An approximate subgradient algorithm for unconstrained nonsmooth, nonconvex optimization Bagirov, Adil; Ganjehlou, Asef Nazari Mathematical Methods of Operations Research, Vol. 67, Issue 2 https://doi.org/10.1007/s00186-007-0186-5	journal	October 2007
On Lipschitz optimization based on gray-box piecewise linearization Griewank, Andreas; Walther, Andrea; Fiege, Sabrina Mathematical Programming, Vol. 158, Issue 1-2 https://doi.org/10.1007/s10107-015-0934-x	journal	July 2015
Derivative-free robust optimization by outer approximations Menickelly, Matt; Wild, Stefan M. Mathematical Programming, Vol. 179, Issue 1-2 https://doi.org/10.1007/s10107-018-1326-9	journal	October 2018
Stochastic Subgradient Method Converges on Tame Functions Davis, Damek; Drusvyatskiy, Dmitriy; Kakade, Sham Foundations of Computational Mathematics, Vol. 20, Issue 1 https://doi.org/10.1007/s10208-018-09409-5	journal	January 2019
A proximal bundle method for nonsmooth nonconvex functions with inexact information Hare, W.; Sagastizábal, C.; Solodov, M. Computational Optimization and Applications, Vol. 63, Issue 1 https://doi.org/10.1007/s10589-015-9762-4	journal	May 2015
Subgradient Method for Nonconvex Nonsmooth Optimization Bagirov, A. M.; Jin, L.; Karmitsa, N. Journal of Optimization Theory and Applications, Vol. 157, Issue 2 https://doi.org/10.1007/s10957-012-0167-6	journal	September 2012
A derivative-free trust-region algorithm for composite nonsmooth optimization Grapiglia, Geovani Nunes; Yuan, Jinyun; Yuan, Ya-xiang Computational and Applied Mathematics, Vol. 35, Issue 2 https://doi.org/10.1007/s40314-014-0201-4	journal	December 2014
Derivative-free optimization methods Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. Acta Numerica, Vol. 28 https://doi.org/10.1017/S0962492919000060	journal	May 2019
Limited memory discrete gradient bundle method for nonsmooth derivative-free optimization Karmitsa, N.; Bagirov, A. M. Optimization, Vol. 61, Issue 12 https://doi.org/10.1080/02331934.2012.687736	journal	December 2012
Survey of Bundle Methods for Nonsmooth Optimization Mäkelä, Marko Optimization Methods and Software, Vol. 17, Issue 1 https://doi.org/10.1080/10556780290027828	journal	January 2002
An adaptive gradient sampling algorithm for non-smooth optimization Curtis, Frank E.; Que, Xiaocun Optimization Methods and Software, Vol. 28, Issue 6 https://doi.org/10.1080/10556788.2012.714781	journal	September 2012
A vector forward mode of automatic differentiation for generalized derivative evaluation Khan, Kamil A.; Barton, Paul I. Optimization Methods and Software, Vol. 30, Issue 6 https://doi.org/10.1080/10556788.2015.1025400	journal	April 2015
A derivative-free 𝒱𝒰-algorithm for convex finite-max problems Hare, Warren; Planiden, Chayne; Sagastizábal, Claudia Optimization Methods and Software, Vol. 35, Issue 3 https://doi.org/10.1080/10556788.2019.1668944	journal	September 2019
A Method for Solving Certain Quadratic Programming Problems Arising in Nonsmooth Optimization Kiwiel, Krzysztof C. IMA Journal of Numerical Analysis, Vol. 6, Issue 2 https://doi.org/10.1093/imanum/6.2.137	journal	January 1986
A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization Burke, James V.; Lewis, Adrian S.; Overton, Michael L. SIAM Journal on Optimization, Vol. 15, Issue 3 https://doi.org/10.1137/030601296	journal	January 2005
Clarke Subgradients of Stratifiable Functions Bolte, Jérôme; Daniilidis, Aris; Lewis, Adrian SIAM Journal on Optimization, Vol. 18, Issue 2 https://doi.org/10.1137/060670080	journal	January 2007
Minimization of Locally Lipschitzian Functions Pang, Jong-Shi; Han, Shih-Ping; Rangaraj, Narayan SIAM Journal on Optimization, Vol. 1, Issue 1 https://doi.org/10.1137/0801006	journal	February 1991
McCormick-Based Relaxations of Algorithms Mitsos, Alexander; Chachuat, Benoît; Barton, Paul I. SIAM Journal on Optimization, Vol. 20, Issue 2 https://doi.org/10.1137/080717341	journal	January 2009
An Invitation to Tame Optimization Ioffe, A. D. SIAM Journal on Optimization, Vol. 19, Issue 4 https://doi.org/10.1137/080722059	journal	January 2009
Benchmarking Derivative-Free Optimization Algorithms Moré, Jorge J.; Wild, Stefan M. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/080724083	journal	January 2009
Computing a Trust Region Step Moré, Jorge J.; Sorensen, D. C. SIAM Journal on Scientific and Statistical Computing, Vol. 4, Issue 3 https://doi.org/10.1137/0904038	journal	September 1983
A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 20, Issue 4 https://doi.org/10.1137/090748408	journal	January 2010
A Redistributed Proximal Bundle Method for Nonconvex Optimization Hare, Warren; Sagastizábal, Claudia SIAM Journal on Optimization, Vol. 20, Issue 5 https://doi.org/10.1137/090754595	journal	January 2010
A Sequential Quadratic Programming Algorithm for Nonconvex, Nonsmooth Constrained Optimization Curtis, Frank E.; Overton, Michael L. SIAM Journal on Optimization, Vol. 22, Issue 2 https://doi.org/10.1137/090780201	journal	January 2012
On the Evaluation Complexity of Composite Function Minimization with Applications to Nonconvex Nonlinear Programming Cartis, Coralia; Gould, Nicholas I. M.; Toint, Philippe L. SIAM Journal on Optimization, Vol. 21, Issue 4 https://doi.org/10.1137/11082381X	journal	October 2011
Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case Garmanjani, R.; Júdice, D.; Vicente, L. N. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/151005683	journal	January 2016
Manifold Sampling for $\ell_1$ Nonconvex Optimization Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/15M1042097	journal	January 2016
Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components Khan, Kamil A.; Larson, Jeffrey; Wild, Stefan M. SIAM Journal on Optimization, Vol. 28, Issue 4 https://doi.org/10.1137/17M114741X	journal	January 2018
Characterizing and Testing Subdifferential Regularity in Piecewise Smooth Optimization Walther, Andrea; Griewank, Andreas SIAM Journal on Optimization, Vol. 29, Issue 2 https://doi.org/10.1137/17M115520X	journal	January 2019
Composite Difference-Max Programs for Modern Statistical Estimation Problems Cui, Ying; Pang, Jong-Shi; Sen, Bodhisattva SIAM Journal on Optimization, Vol. 28, Issue 4 https://doi.org/10.1137/18M117337X	journal	January 2018
Trust-Region Methods for the Derivative-Free Optimization of Nonsmooth Black-Box Functions Liuzzi, Giampaolo; Lucidi, Stefano; Rinaldi, Francesco SIAM Journal on Optimization, Vol. 29, Issue 4 https://doi.org/10.1137/19M125772X	journal	January 2019
Evaluating an element of the Clarke generalized Jacobian of a composite piecewise differentiable function Khan, Kamil A.; Barton, Paul I. ACM Transactions on Mathematical Software, Vol. 39, Issue 4 https://doi.org/10.1145/2491491.2491493	journal	July 2013
Improving the Flexibility and Robustness of Model-based Derivative-free Optimization Solvers Cartis, Coralia; Fiala, Jan; Marteau, Benjamin ACM Transactions on Mathematical Software, Vol. 45, Issue 3 https://doi.org/10.1145/3338517	journal	September 2019
Approximating Subdifferentials by Random Sampling of Gradients Burke, J. V.; Lewis, A. S.; Overton, M. L. Mathematics of Operations Research, Vol. 27, Issue 3 https://doi.org/10.1287/moor.27.3.567.317	journal	August 2002

Cited By (1)

Derivative-Free Optimization of a Rapid-Cycling Synchrotron Eldred, Jeffrey S.; Larson, Jeffrey; Padidar, Misha arXiv https://doi.org/10.48550/arxiv.2108.04774	text	January 2021

Similar Records

Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components

Journal Article · Wed Oct 24 20:00:00 EDT 2018 · SIAM Journal on Optimization · OSTI ID:1491737

A proximal trust-region method for nonsmooth optimization with inexact function and gradient evaluations

Journal Article · Sun Dec 25 19:00:00 EST 2022 · Mathematical Programming · OSTI ID:2311742

Efficient proximal subproblem solvers for a nonsmooth trust-region method

Journal Article · Fri Jan 03 19:00:00 EST 2025 · Computational Optimization and Applications · OSTI ID:2505123

Related Subjects

97 MATHEMATICS AND COMPUTING
gradient sampling
manifold sampling
nonsmooth optimization

Manifold Sampling for Optimizing Nonsmooth Nonconvex Compositions

Citation Formats

References (46)

Cited By (1)

Similar Records

Related Subjects