Structure-aware methods for expensive derivative-free nonsmooth composite optimization

Larson, Jeffrey; Menickelly, Matt

doi:10.1007/s12532-023-00245-5

Structure-aware methods for expensive derivative-free nonsmooth composite optimization

Journal Article · Sat Aug 19 00:00:00 EDT 2023 · Mathematical Programming Computation

DOI:https://doi.org/10.1007/s12532-023-00245-5· OSTI ID:2377274

Larson, Jeffrey ^[1]; Menickelly, Matt ^[1]

Argonne National Laboratory (ANL), Argonne, IL (United States)

We present new methods for solving a broad class of bound-constrained nonsmooth composite minimization problems. These methods are specially designed for objectives that are some known mapping of outputs from a computationally expensive function. We provide accompanying implementations of these methods: in particular, a novel manifold sampling algorithm (MS-P) with subproblems that are in a sense primal versions of the dual problems solved by previous manifold sampling methods and a method (GOOMBAH) that employs more difficult optimization subproblems. For these two methods, we provide rigorous convergence analysis and guarantees. We demonstrate extensive testing of these methods. Open-source implementations of the methods developed in this manuscript can be found at https://github.com/POptUS/ IBCDFO/.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 2377274

Journal Information:: Mathematical Programming Computation, Journal Name: Mathematical Programming Computation Journal Issue: 1 Vol. 16; ISSN 1867-2949

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (43)

Introduction to Piecewise Differentiable Equations Scholtes, Stefan SpringerBriefs in Optimization https://doi.org/10.1007/978-1-4614-4340-7	book	January 2012
Particle Accelerator Physics Wiedemann, Helmut Graduate Texts in Physics https://doi.org/10.1007/978-3-319-18317-6	book	January 2015
Derivative-Free and Blackbox Optimization Audet, Charles; Hare, Warren Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/978-3-319-68913-5	book	January 2017
Variational Analysis Rockafellar, R. Tyrrell; Wets, Roger J. B. Grundlehren der mathematischen Wissenschaften https://doi.org/10.1007/978-3-642-02431-3	book	January 1998
An algorithm for composite nonsmooth optimization problems Womersley, R. S.; Fletcher, R. Journal of Optimization Theory and Applications, Vol. 48, Issue 3 https://doi.org/10.1007/bf00940574	journal	March 1986
Conditions for convergence of trust region algorithms for nonsmooth optimization Yuan, Y. Mathematical Programming, Vol. 31, Issue 2 https://doi.org/10.1007/bf02591750	journal	June 1985
On the superlinear convergence of a trust region algorithm for nonsmooth optimization Yuan, Y. Mathematical Programming, Vol. 31, Issue 3 https://doi.org/10.1007/bf02591949	journal	October 1985
Second order corrections for non-differentiable optimization Fletcher, R. Numerical Analysis https://doi.org/10.1007/bfb0093151	book	January 1982
An approximate subgradient algorithm for unconstrained nonsmooth, nonconvex optimization Bagirov, Adil; Ganjehlou, Asef Nazari Mathematical Methods of Operations Research, Vol. 67, Issue 2 https://doi.org/10.1007/s00186-007-0186-5	journal	October 2007
Nonsmooth optimization via quasi-Newton methods Lewis, Adrian S.; Overton, Michael L. Mathematical Programming, Vol. 141, Issue 1-2 https://doi.org/10.1007/s10107-012-0514-2	journal	February 2012
Composite proximal bundle method Sagastizábal, Claudia Mathematical Programming, Vol. 140, Issue 1 https://doi.org/10.1007/s10107-012-0600-5	journal	October 2012
Derivative-free robust optimization by outer approximations Menickelly, Matt; Wild, Stefan M. Mathematical Programming, Vol. 179, Issue 1-2 https://doi.org/10.1007/s10107-018-1326-9	journal	October 2018
A Geometric Integration Approach to Nonsmooth, Nonconvex Optimisation Riis, Erlend S.; Ehrhardt, Matthias J.; Quispel, G. R. W. Foundations of Computational Mathematics, Vol. 22, Issue 5 https://doi.org/10.1007/s10208-020-09489-2	journal	July 2021
A proximal bundle method for nonsmooth nonconvex functions with inexact information Hare, W.; Sagastizábal, C.; Solodov, M. Computational Optimization and Applications, Vol. 63, Issue 1 https://doi.org/10.1007/s10589-015-9762-4	journal	May 2015
Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization Bagirov, A. M.; Karasözen, B.; Sezer, M. Journal of Optimization Theory and Applications, Vol. 137, Issue 2 https://doi.org/10.1007/s10957-007-9335-5	journal	December 2007
Subgradient Method for Nonconvex Nonsmooth Optimization Bagirov, A. M.; Jin, L.; Karmitsa, N. Journal of Optimization Theory and Applications, Vol. 157, Issue 2 https://doi.org/10.1007/s10957-012-0167-6	journal	September 2012
Derivative-free optimization of a rapid-cycling synchrotron Eldred, Jeffrey S.; Larson, Jeffrey; Padidar, Misha Optimization and Engineering, Vol. 24, Issue 2 https://doi.org/10.1007/s11081-022-09733-4	journal	June 2022
A derivative-free trust-region algorithm for composite nonsmooth optimization Grapiglia, Geovani Nunes; Yuan, Jinyun; Yuan, Ya-xiang Computational and Applied Mathematics, Vol. 35, Issue 2 https://doi.org/10.1007/s40314-014-0201-4	journal	December 2014
Limited memory discrete gradient bundle method for nonsmooth derivative-free optimization Karmitsa, N.; Bagirov, A. M. Optimization, Vol. 61, Issue 12 https://doi.org/10.1080/02331934.2012.687736	journal	December 2012
Survey of Bundle Methods for Nonsmooth Optimization Mäkelä, Marko Optimization Methods and Software, Vol. 17, Issue 1 https://doi.org/10.1080/10556780290027828	journal	January 2002
An adaptive gradient sampling algorithm for non-smooth optimization Curtis, Frank E.; Que, Xiaocun Optimization Methods and Software, Vol. 28, Issue 6 https://doi.org/10.1080/10556788.2012.714781	journal	September 2012
A derivative-free 𝒱𝒰-algorithm for convex finite-max problems Hare, Warren; Planiden, Chayne; Sagastizábal, Claudia Optimization Methods and Software, Vol. 35, Issue 3 https://doi.org/10.1080/10556788.2019.1668944	journal	September 2019
A Robust Gradient Sampling Algorithm for Nonsmooth, Nonconvex Optimization Burke, James V.; Lewis, Adrian S.; Overton, Michael L. SIAM Journal on Optimization, Vol. 15, Issue 3 https://doi.org/10.1137/030601296	journal	January 2005
Convergence of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 18, Issue 2 https://doi.org/10.1137/050639673	journal	January 2007
Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points Conn, Andrew R.; Scheinberg, Katya; Vicente, Luís N. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/060673424	journal	January 2009
Restricted Step and Levenberg–Marquardt Techniques in Proximal Bundle Methods for Nonconvex Nondifferentiable Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 6, Issue 1 https://doi.org/10.1137/0806013	journal	February 1996
Benchmarking Derivative-Free Optimization Algorithms Moré, Jorge J.; Wild, Stefan M. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/080724083	journal	January 2009
Censored Discrete Linear $l_1 $ Approximation Womersley, R. S. SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 1 https://doi.org/10.1137/0907008	journal	January 1986
A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 20, Issue 4 https://doi.org/10.1137/090748408	journal	January 2010
A Redistributed Proximal Bundle Method for Nonconvex Optimization Hare, Warren; Sagastizábal, Claudia SIAM Journal on Optimization, Vol. 20, Issue 5 https://doi.org/10.1137/090754595	journal	January 2010
Introduction to Derivative-Free Optimization Conn, Andrew R.; Scheinberg, Katya; Vicente, Luis N. https://doi.org/10.1137/1.9780898718768	book	January 2009
Trust Region Methods Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe L. MOS-SIAM Series on Optimization https://doi.org/10.1137/1.9780898719857	book	January 2000
Chapter 40: POUNDERS in TAO: Solving Derivative-Free Nonlinear Least-Squares Problems with POUNDERS Wild, Stefan M.; Terlaky, Tamás; Anjos, Miguel F. Advances and Trends in Optimization with Engineering Applications https://doi.org/10.1137/1.9781611974683.ch40	book	April 2017
Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case Garmanjani, R.; Júdice, D.; Vicente, L. N. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/151005683	journal	January 2016
Manifold Sampling for $\ell_1$ Nonconvex Optimization Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/15M1042097	journal	January 2016
Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components Khan, Kamil A.; Larson, Jeffrey; Wild, Stefan M. SIAM Journal on Optimization, Vol. 28, Issue 4 https://doi.org/10.1137/17m114741x	journal	January 2018
Trust-Region Methods for the Derivative-Free Optimization of Nonsmooth Black-Box Functions Liuzzi, Giampaolo; Lucidi, Stefano; Rinaldi, Francesco SIAM Journal on Optimization, Vol. 29, Issue 4 https://doi.org/10.1137/19m125772x	journal	January 2019
Tuning Multigrid Methods with Robust Optimization and Local Fourier Analysis Brown, Jed; He, Yunhui; MacLachlan, Scott SIAM Journal on Scientific Computing, Vol. 43, Issue 1 https://doi.org/10.1137/19m1308669	journal	January 2021
Manifold Sampling for Optimizing Nonsmooth Nonconvex Compositions Larson, Jeffrey; Menickelly, Matt; Zhou, Baoyu SIAM Journal on Optimization, Vol. 31, Issue 4 https://doi.org/10.1137/20M1378089	journal	January 2021
CONOPT—A Large-Scale GRG Code Drud, Arne Stolbjerg ORSA Journal on Computing, Vol. 6, Issue 2 https://doi.org/10.1287/ijoc.6.2.207	journal	May 1994
Approximating Subdifferentials by Random Sampling of Gradients Burke, J. V.; Lewis, A. S.; Overton, M. L. Mathematics of Operations Research, Vol. 27, Issue 3 https://doi.org/10.1287/moor.27.3.567.317	journal	August 2002
Error Analysis of Surrogate Models Constructed through Operations on Sub-models Chen, Yiwen; Jarry-Bolduc, Gabriel; Hare, Warren arXiv https://doi.org/10.48550/ARXIV.2112.08411	text	January 2021
Harnessing structure in composite nonsmooth minimization Bareilles, Gilles; Iutzeler, Franck; Malick, Jérôme arXiv https://doi.org/10.48550/arxiv.2206.15053	text	January 2022

Similar Records

Manifold Sampling for Optimizing Nonsmooth Nonconvex Compositions

Journal Article · Thu Oct 28 00:00:00 EDT 2021 · SIAM Journal on Optimization · OSTI ID:1839939

A Smoothed Augmented Lagrangian Framework for Convex Optimization with Nonsmooth Constraints

Journal Article · Sat Jun 21 00:00:00 EDT 2025 · Journal of Scientific Computing · OSTI ID:3010461

Model Predictive Control of a Wave Energy Converter Using Duality Techniques

Conference · Thu Aug 29 00:00:00 EDT 2019 · OSTI ID:1569435

Related Subjects

97 MATHEMATICS AND COMPUTING
Derivative-free optimization
composite optimization
continuous selections
manifold sampling
nonsmooth optimization

Structure-aware methods for expensive derivative-free nonsmooth composite optimization

Citation Formats

References (43)

Similar Records

Related Subjects