Derivative-free optimization methods

Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M.

doi:10.1017/S0962492919000060

Title: Derivative-free optimization methods

Abstract

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

Authors:

Larson, Jeffrey ^[1]; Menickelly, Matt ^[1]; Wild, Stefan M. ^[1]

Argonne National Lab. (ANL), Argonne, IL (United States)

Publication Date:: Wed May 01 00:00:00 EDT 2019

Research Org.:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1545343

Grant/Contract Number:: AC02-06CH11357

Resource Type:: Accepted Manuscript

Journal Name:: Acta Numerica

Additional Journal Information:: Journal Volume: 28; Journal Issue: 2010; Journal ID: ISSN 0962-4929

Publisher:: Cambridge University Press

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Optimization

Citation Formats


                    Larson, Jeffrey, Menickelly, Matt, and Wild, Stefan M. Derivative-free optimization methods.  United States: N. p., 2019. 
Web.  doi:10.1017/S0962492919000060.

Copy to clipboard


                    Larson, Jeffrey, Menickelly, Matt, & Wild, Stefan M. Derivative-free optimization methods.  United States.  https://doi.org/10.1017/S0962492919000060

Copy to clipboard


                    Larson, Jeffrey, Menickelly, Matt, and Wild, Stefan M. Wed .  
"Derivative-free optimization methods".  United States.  https://doi.org/10.1017/S0962492919000060.  https://www.osti.gov/servlets/purl/1545343.

Copy to clipboard


                    
@article{osti_1545343,

  title        = {Derivative-free optimization methods},

  author       = {Larson, Jeffrey and Menickelly, Matt and Wild, Stefan M.},

  abstractNote = {In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.},

  doi          = {10.1017/S0962492919000060},

  journal      = {Acta Numerica},

  number       = 2010,

  volume       = 28,

  place        = {United States},

  year         = {Wed May 01 00:00:00 EDT 2019},

  month        = {Wed May 01 00:00:00 EDT 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1017/S0962492919000060

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 139 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Sequential Design and Analysis of High-Accuracy and Low-Accuracy Computer Codes
journal, August 2012

Xiong, Shifeng; Qian, Peter Z. G.; Wu, C. F. Jeff
Technometrics, Vol. 55, Issue 1
DOI: 10.1080/00401706.2012.723572

Manifold Sampling for $\ell_1$ Nonconvex Optimization
journal, January 2016

Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M.
SIAM Journal on Optimization, Vol. 26, Issue 4
DOI: 10.1137/15M1042097

Mesh adaptive direct search algorithms for mixed variable optimization
journal, June 2008

Abramson, Mark A.; Audet, Charles; Chrissis, James W.
Optimization Letters, Vol. 3, Issue 1
DOI: 10.1007/s11590-008-0089-2

A trust-region derivative-free algorithm for constrained optimization
journal, April 2015

Conejo, P. D.; Karas, E. W.; Pedroso, L. G.
Optimization Methods and Software, Vol. 30, Issue 6
DOI: 10.1080/10556788.2015.1026968

Convergence estimates for iterative minimization methods
journal, January 1974

Karmanov, V. G.
USSR Computational Mathematics and Mathematical Physics, Vol. 14, Issue 1
DOI: 10.1016/0041-5553(74)90131-1

Derivative-Free and Blackbox Optimization
book, January 2017

Audet, Charles; Hare, Warren
Springer Series in Operations Research and Financial Engineering
DOI: 10.1007/978-3-319-68913-5

Algorithm 897: VTDIRECT95: Serial and parallel codes for the global optimization algorithm direct
journal, July 2009

He, Jian; Watson, Layne T.; Sosonkina, Masha
ACM Transactions on Mathematical Software, Vol. 36, Issue 3
DOI: 10.1145/1527286.1527291

A Pattern Search Filter Method for Nonlinear Programming without Derivatives
journal, January 2004

Audet, Charles; Dennis, J. E.
SIAM Journal on Optimization, Vol. 14, Issue 4
DOI: 10.1137/S105262340138983X

On the Local Convergence of Pattern Search
journal, January 2003

Dolan, Elizabeth D.; Lewis, Robert Michael; Torczon, Virginia
SIAM Journal on Optimization, Vol. 14, Issue 2
DOI: 10.1137/S1052623400374495

Global Convergence of Radial Basis Function Trust-Region Algorithms for Derivative-Free Optimization
journal, January 2013

Wild, Stefan M.; Shoemaker, Christine
SIAM Review, Vol. 55, Issue 2
DOI: 10.1137/120902434

An active-set trust-region method for derivative-free nonlinear bound-constrained optimization
journal, October 2011

Gratton, Serge; Toint, Philippe L.; Tröltzsch, Anke
Optimization Methods and Software, Vol. 26, Issue 4-5
DOI: 10.1080/10556788.2010.549231

Multiobjective Optimization Through a Series of Single-Objective Formulations
journal, January 2008

Audet, Charles; Savard, Gilles; Zghal, Walid
SIAM Journal on Optimization, Vol. 19, Issue 1
DOI: 10.1137/060677513

A Merit Function Approach for Direct Search
journal, January 2014

Gratton, S.; Vicente, L. N.
SIAM Journal on Optimization, Vol. 24, Issue 4
DOI: 10.1137/130917661

A View of Unconstrained Minimization Algorithms that Do Not Require Derivatives
journal, June 1975

Powell, M. J. D.
ACM Transactions on Mathematical Software, Vol. 1, Issue 2
DOI: 10.1145/355637.355638

Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers
journal, November 1999

Kleinman, Nathan L.; Spall, James C.; Naiman, Daniel Q.
Management Science, Vol. 45, Issue 11
DOI: 10.1287/mnsc.45.11.1570

Self-Correcting Geometry in Model-Based Algorithms for Derivative-Free Unconstrained Optimization
journal, January 2010

Scheinberg, K.; Toint, Ph. L.
SIAM Journal on Optimization, Vol. 20, Issue 6
DOI: 10.1137/090748536

A Derivative-Free Algorithm for Linearly Constrained Finite Minimax Problems
journal, January 2006

Liuzzi, G.; Lucidi, S.; Sciandrone, M.
SIAM Journal on Optimization, Vol. 16, Issue 4
DOI: 10.1137/040615821

Robust optimization of noisy blackbox problems using the Mesh Adaptive Direct Search algorithm
journal, January 2018

Audet, Charles; Ihaddadene, Amina; Le Digabel, Sébastien
Optimization Letters, Vol. 12, Issue 4
DOI: 10.1007/s11590-017-1226-6

Some aspects of the sequential design of experiments
journal, January 1952

Robbins, Herbert
Bulletin of the American Mathematical Society, Vol. 58, Issue 5
DOI: 10.1090/S0002-9904-1952-09620-8

On the local convergence of a derivative-free algorithm for least-squares minimization
journal, October 2010

Zhang, Hongchao; Conn, Andrew R.
Computational Optimization and Applications, Vol. 51, Issue 2
DOI: 10.1007/s10589-010-9367-x

A trust-region-based derivative free algorithm for mixed integer programming
journal, April 2014

Newby, Eric; Ali, M. M.
Computational Optimization and Applications, Vol. 60, Issue 1
DOI: 10.1007/s10589-014-9660-1

Numerical Analysis of $\mathcal{V}\mathcal{U}$-Decomposition, $\mathcal{U}$-Gradient, and $\mathcal{U}$-Hessian Approximations
journal, January 2014

Hare, W.
SIAM Journal on Optimization, Vol. 24, Issue 4
DOI: 10.1137/130933691

On Choosing Parameters in Retrospective-Approximation Algorithms for Stochastic Root Finding and Simulation Optimization
journal, August 2010

Pasupathy, Raghu
Operations Research, Vol. 58, Issue 4-part-1
DOI: 10.1287/opre.1090.0773

Random Gradient-Free Minimization of Convex Functions
journal, November 2015

Nesterov, Yurii; Spokoiny, Vladimir
Foundations of Computational Mathematics, Vol. 17, Issue 2
DOI: 10.1007/s10208-015-9296-2

A grid algorithm for bound constrained optimization of noisy functions
journal, January 1995

Elster, Clemens; Neumaier, Arnold
IMA Journal of Numerical Analysis, Vol. 15, Issue 4
DOI: 10.1093/imanum/15.4.585

Efficient calculation of regular simplex gradients
journal, January 2019

Coope, Ian; Tappenden, Rachael
Computational Optimization and Applications, Vol. 72, Issue 3
DOI: 10.1007/s10589-019-00063-3

Superlinear Convergence and Implicit Filtering
journal, January 2000

Choi, T. D.; Kelley, C. T.
SIAM Journal on Optimization, Vol. 10, Issue 4
DOI: 10.1137/S1052623499354096

Estimating Computational Noise
journal, January 2011

Moré, Jorge J.; Wild, Stefan M.
SIAM Journal on Scientific Computing, Vol. 33, Issue 3
DOI: 10.1137/100786125

A projected derivative-free algorithm for nonlinear equations with convex constraints
journal, July 2012

La Cruz, William
Optimization Methods and Software, Vol. 29, Issue 1
DOI: 10.1080/10556788.2012.721129

Sequential Search: A Method for Solving Constrained Optimization Problems
journal, January 1965

Glass, H.; Cooper, L.
Journal of the ACM, Vol. 12, Issue 1
DOI: 10.1145/321250.321256

Asymptotically efficient adaptive allocation rules
journal, March 1985

Lai, T. L.; Robbins, Herbert
Advances in Applied Mathematics, Vol. 6, Issue 1
DOI: 10.1016/0196-8858(85)90002-8

Worst case complexity of direct search under convexity
journal, November 2014

Dodangeh, M.; Vicente, L. N.
Mathematical Programming, Vol. 155, Issue 1-2
DOI: 10.1007/s10107-014-0847-0

Optimization with hidden constraints and embedded Monte Carlo computations
journal, December 2015

Chen, Xiaojun; Kelley, C. T.
Optimization and Engineering, Vol. 17, Issue 1
DOI: 10.1007/s11081-015-9302-1

Multidimensional Stochastic Approximation Methods
journal, December 1954

Blum, Julius R.
The Annals of Mathematical Statistics, Vol. 25, Issue 4
DOI: 10.1214/aoms/1177728659

A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization
journal, January 2010

Kiwiel, Krzysztof C.
SIAM Journal on Optimization, Vol. 20, Issue 4
DOI: 10.1137/090748408

Pattern Search Algorithms for Bound Constrained Minimization
journal, January 1999

Lewis, Robert Michael; Torczon, Virginia
SIAM Journal on Optimization, Vol. 9, Issue 4
DOI: 10.1137/S1052623496300507

Cokriging-Based Sequential Design Strategies Using Fast Cross-Validation Techniques for Multi-Fidelity Computer Codes
journal, July 2015

Le Gratiet, Loic; Cannamela, Claire
Technometrics, Vol. 57, Issue 3
DOI: 10.1080/00401706.2014.928233

Kernel-based methods for Bandit convex optimization
conference, January 2017

Bubeck, Sébastien; Lee, Yin Tat; Eldan, Ronen
Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing - STOC 2017
DOI: 10.1145/3055399.3055403

Direct search algorithm for bilevel programming problems
journal, September 2009

Mersha, Ayalew Getachew; Dempe, Stephan
Computational Optimization and Applications, Vol. 49, Issue 1
DOI: 10.1007/s10589-009-9295-9

A method for the solution of certain non-linear problems in least squares
journal, January 1944

Levenberg, Kenneth
Quarterly of Applied Mathematics, Vol. 2, Issue 2
DOI: 10.1090/qam/10666

Spectral residual method without gradient information for solving large-scale nonlinear systems of equations
journal, April 2006

La Cruz, William; Martínez, José Mario; Raydan, Marcos
Mathematics of Computation, Vol. 75, Issue 255
DOI: 10.1090/S0025-5718-06-01840-0

The solution of variational and boundary value problems by the method of local variations
journal, January 1966

Banichuk, N. V.; Petrov, V. M.; Chernous'ko, F. L.
USSR Computational Mathematics and Mathematical Physics, Vol. 6, Issue 6
DOI: 10.1016/0041-5553(66)90158-3

Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization
journal, September 2006

Gray, Genetha A.; Kolda, Tamara G.
ACM Transactions on Mathematical Software, Vol. 32, Issue 3
DOI: 10.1145/1163641.1163647

Compositions of convex functions and fully linear models
journal, February 2017

Hare, W.
Optimization Letters, Vol. 11, Issue 7
DOI: 10.1007/s11590-017-1117-x

Global Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms
journal, July 2011

Wild, Stefan M.; Shoemaker, Christine
SIAM Journal on Optimization, Vol. 21, Issue 3
DOI: 10.1137/09074927X

Sample mean based index policies by O (log n ) regret for the multi-armed bandit problem
journal, December 1995

Agrawal, Rajeev
Advances in Applied Probability, Vol. 27, Issue 4
DOI: 10.2307/1427934

Variable-Number Sample-Path Optimization
journal, July 2007

Deng, Geng; Ferris, Michael C.
Mathematical Programming, Vol. 117, Issue 1-2
DOI: 10.1007/s10107-007-0164-y

A Derivative-Free Trust-Region Method for Biobjective Optimization
journal, January 2014

Ryu, Jong-hyun; Kim, Sujin
SIAM Journal on Optimization, Vol. 24, Issue 1
DOI: 10.1137/120864738

Conditions for convergence of trust region algorithms for nonsmooth optimization
journal, June 1985

Yuan, Y.
Mathematical Programming, Vol. 31, Issue 2
DOI: 10.1007/BF02591750

Convergence Results for Generalized Pattern Search Algorithms are Tight
journal, June 2004

Audet, Charles
Optimization and Engineering, Vol. 5, Issue 2
DOI: 10.1023/B:OPTE.0000033370.66768.a9

Constrained derivative-free optimization on thin domains
journal, July 2012

Martínez, J. M.; Sobral, F. N. C.
Journal of Global Optimization, Vol. 56, Issue 3
DOI: 10.1007/s10898-012-9944-x

Direct search based on probabilistic feasible descent for bound and linearly constrained problems
journal, January 2019

Gratton, S.; Royer, C. W.; Vicente, L. N.
Computational Optimization and Applications, Vol. 72, Issue 3
DOI: 10.1007/s10589-019-00062-4

Stochastic online optimization. Single-point and multi-point non-linear multi-armed bandits. Convex and strongly-convex case
journal, February 2017

Gasnikov, A. V.; Krymova, E. A.; Lagunovskaya, A. A.
Automation and Remote Control, Vol. 78, Issue 2
DOI: 10.1134/S0005117917020035

A Derivative-Free Algorithm for Inequality Constrained Nonlinear Programming via Smoothing of an $\ell_\infty$ Penalty Function
journal, January 2009

Liuzzi, G.; Lucidi, S.
SIAM Journal on Optimization, Vol. 20, Issue 1
DOI: 10.1137/070711451

Design and implementation of a massively parallel version of DIRECT
journal, October 2007

He, Jian; Verstak, Alex; Watson, Layne T.
Computational Optimization and Applications, Vol. 40, Issue 2
DOI: 10.1007/s10589-007-9092-2

Approximation Methods which Converge with Probability one
journal, June 1954

Blum, Julius R.
The Annals of Mathematical Statistics, Vol. 25, Issue 2
DOI: 10.1214/aoms/1177728794

Derivative-Free Methods for Mixed-Integer Constrained Optimization Problems
journal, July 2014

Liuzzi, Giampaolo; Lucidi, Stefano; Rinaldi, Francesco
Journal of Optimization Theory and Applications, Vol. 164, Issue 3
DOI: 10.1007/s10957-014-0617-4

MISO: mixed-integer surrogate optimization framework
journal, June 2015

Müller, Juliane
Optimization and Engineering, Vol. 17, Issue 1
DOI: 10.1007/s11081-015-9281-2

Asynchronous Parallel Pattern Search for Nonlinear Optimization
journal, January 2001

Hough, Patricia D.; Kolda, Tamara G.; Torczon, Virginia J.
SIAM Journal on Scientific Computing, Vol. 23, Issue 1
DOI: 10.1137/S1064827599365823

On the convergence of a wide range of trust region methods for unconstrained optimization
journal, September 2009

Powell, M. J. D.
IMA Journal of Numerical Analysis, Vol. 30, Issue 1
DOI: 10.1093/imanum/drp021

Parallel Space Decomposition of the Mesh Adaptive Direct Search Algorithm
journal, January 2008

Audet, Charles; Dennis, J. E.; Digabel, Sébastien Le
SIAM Journal on Optimization, Vol. 19, Issue 3
DOI: 10.1137/070707518

Incorporating minimum Frobenius norm models in direct search
journal, August 2009

Custódio, A. L.; Rocha, H.; Vicente, L. N.
Computational Optimization and Applications, Vol. 46, Issue 2
DOI: 10.1007/s10589-009-9283-0

A batch, derivative-free algorithm for finding multiple local minima
journal, October 2015

Larson, Jeffrey; Wild, Stefan M.
Optimization and Engineering, Vol. 17, Issue 1
DOI: 10.1007/s11081-015-9289-7

Generalized pattern searches with derivative information
journal, November 2003

Abramson, Mark A.; Audet, Charles; Dennis, J. E.
Mathematical Programming, Vol. 100, Issue 1
DOI: 10.1007/s10107-003-0484-5

A derivative-free trust-region algorithm for composite nonsmooth optimization
journal, December 2014

Grapiglia, Geovani Nunes; Yuan, Jinyun; Yuan, Ya-xiang
Computational and Applied Mathematics, Vol. 35, Issue 2
DOI: 10.1007/s40314-014-0201-4

Pattern search ranking and selection algorithms for mixed variable simulation-based optimization
journal, November 2009

Sriver, Todd A.; Chrissis, James W.; Abramson, Mark A.
European Journal of Operational Research, Vol. 198, Issue 3
DOI: 10.1016/j.ejor.2008.10.020

Derivative-free optimization: a review of algorithms and comparison of software implementations
journal, July 2012

Rios, Luis Miguel; Sahinidis, Nikolaos V.
Journal of Global Optimization, Vol. 56, Issue 3
DOI: 10.1007/s10898-012-9951-y

Mesh Adaptive Direct Search Algorithms for Constrained Optimization
journal, January 2006

Audet, Charles; Dennis, J. E.
SIAM Journal on Optimization, Vol. 17, Issue 1
DOI: 10.1137/040603371

A trust-region framework for managing the use of approximation models in optimization
journal, February 1998

Alexandrov, N. M.; Dennis, J. E.; Lewis, R. M.
Structural Optimization, Vol. 15, Issue 1
DOI: 10.1007/BF01197433

On the Oracle Complexity of First-Order and Derivative-Free Algorithms for Smooth Nonconvex Minimization
journal, January 2012

Cartis, Coralia; Gould, Nicholas I. M.; Toint, Philippe L.
SIAM Journal on Optimization, Vol. 22, Issue 1
DOI: 10.1137/100812276

A stochastic method for global optimization
journal, December 1982

Boender, C. G. E.; Rinnooy Kan, A. H. G.; Timmer, G. T.
Mathematical Programming, Vol. 22, Issue 1
DOI: 10.1007/BF01581033

A class of derivative-free methods for large-scale nonlinear monotone equations
journal, January 2011

Li, Q.; Li, D. -H.
IMA Journal of Numerical Analysis, Vol. 31, Issue 4
DOI: 10.1093/imanum/drq015

A Convergent Variant of the Nelder–Mead Algorithm
journal, April 2002

Price, C. J.; Coope, I. D.; Byatt, D.
Journal of Optimization Theory and Applications, Vol. 113, Issue 1
DOI: 10.1023/A:1014849028575

VXQR: derivative-free unconstrained optimization based on QR factorizations
journal, September 2010

Neumaier, Arnold; Fendl, Hannes; Schilly, Harald
Soft Computing, Vol. 15, Issue 11
DOI: 10.1007/s00500-010-0652-5

Geometry of interpolation sets in derivative free optimization
journal, December 2006

Conn, A. R.; Scheinberg, K.; Vicente, Luís N.
Mathematical Programming, Vol. 111, Issue 1-2
DOI: 10.1007/s10107-006-0073-5

On the Number of Iterations of Piyavskii's Global Optimization Algorithm
journal, May 1991

Hansen, P.; Jaumard, B.; Lu, S. -H.
Mathematics of Operations Research, Vol. 16, Issue 2
DOI: 10.1287/moor.16.2.334

A derivative-free line search and global convergence of Broyden-like method for nonlinear equations
journal, January 2000

Li, Dong-Hui; Fukushima, Masao
Optimization Methods and Software, Vol. 13, Issue 3
DOI: 10.1080/10556780008805782

Beyond symmetric Broyden for updating quadratic models in minimization without derivatives
journal, January 2012

Powell, M. J. D.
Mathematical Programming, Vol. 138, Issue 1-2
DOI: 10.1007/s10107-011-0510-y

Performance Modeling and Analysis of a Massively Parallel Direct—Part 1
journal, February 2009

Jian He, ; Verstak, Alex; Watson, L. T.
The International Journal of High Performance Computing Applications, Vol. 23, Issue 1
DOI: 10.1177/1094342008098462

A subclass of generating set search with convergence to second-order stationary points
journal, June 2013

Abramson, Mark A.; Frimannslund, Lennart; Steihaug, Trond
Optimization Methods and Software, Vol. 29, Issue 5
DOI: 10.1080/10556788.2013.808198

Performance Modeling and Analysis of a Massively Parallel Direct—Part 2
journal, February 2009

Jian He, ; Verstak, Alex; Sosonkina, M.
The International Journal of High Performance Computing Applications, Vol. 23, Issue 1
DOI: 10.1177/1094342008098463

Direct search methods: then and now
journal, December 2000

Lewis, Robert Michael; Torczon, Virginia; Trosset, Michael W.
Journal of Computational and Applied Mathematics, Vol. 124, Issue 1-2
DOI: 10.1016/S0377-0427(00)00423-4

OrthoMADS: A Deterministic MADS Instance with Orthogonal Directions
journal, January 2009

Abramson, Mark A.; Audet, Charles; Dennis,, J. E.
SIAM Journal on Optimization, Vol. 20, Issue 2
DOI: 10.1137/080716980

Constrained optimization involving expensive function evaluations: A sequential approach
journal, January 2005

Brekelmans, Ruud; Driessen, Lonneke; Hamers, Herbert
European Journal of Operational Research, Vol. 160, Issue 1
DOI: 10.1016/j.ejor.2003.10.009

Boosters: A Derivative-Free Algorithm Based on Radial Basis Functions
journal, January 2009

Oeuvray, R.; Bierlaire, M.
International Journal of Modelling and Simulation, Vol. 29, Issue 1
DOI: 10.1080/02286203.2009.11442507

Two-Point Step Size Gradient Methods
journal, January 1988

Barzilai, Jonathan; Borwein, Jonathan M.
IMA Journal of Numerical Analysis, Vol. 8, Issue 1
DOI: 10.1093/imanum/8.1.141

Stochastic Nelder–Mead simplex method – A new globally convergent direct search method for simulation optimization
journal, August 2012

Chang, Kuo-Hao
European Journal of Operational Research, Vol. 220, Issue 3
DOI: 10.1016/j.ejor.2012.02.028

Stationarity Results for Generating Set Search for Linearly Constrained Optimization
journal, January 2007

Kolda, Tamara G.; Lewis, Robert Michael; Torczon, Virginia
SIAM Journal on Optimization, Vol. 17, Issue 4
DOI: 10.1137/S1052623403433638

An Implicit Filtering Algorithm for Optimization of Functions with Many Local Minima
journal, May 1995

Gilmore, P.; Kelley, C. T.
SIAM Journal on Optimization, Vol. 5, Issue 2
DOI: 10.1137/0805015

Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods
journal, January 2003

Kolda, Tamara G.; Lewis, Robert Michael; Torczon, Virginia
SIAM Review, Vol. 45, Issue 3
DOI: 10.1137/S003614450242889

Inexact Restoration Method for Derivative-Free Optimization with Smooth Constraints
journal, January 2013

Bueno, L. F.; Friedlander, A.; Martínez, J. M.
SIAM Journal on Optimization, Vol. 23, Issue 2
DOI: 10.1137/110856253

A Derivative-Free Approach to Constrained Multiobjective Nonsmooth Optimization
journal, January 2016

Liuzzi, G.; Lucidi, S.; Rinaldi, F.
SIAM Journal on Optimization, Vol. 26, Issue 4
DOI: 10.1137/15M1037810

Global convergence and stabilization of unconstrained minimization methods without derivatives
journal, March 1988

Grippo, L.; Lampariello, F.; Lucidi, S.
Journal of Optimization Theory and Applications, Vol. 56, Issue 3
DOI: 10.1007/BF00939550

A derivative-free comirror algorithm for convex optimization
journal, October 2014

Bauschke, Heinz H.; Hare, Warren L.; Moursi, Walaa M.
Optimization Methods and Software, Vol. 30, Issue 4
DOI: 10.1080/10556788.2014.968158

An Algorithm for Least-Squares Estimation of Nonlinear Parameters
journal, June 1963

Marquardt, Donald W.
Journal of the Society for Industrial and Applied Mathematics, Vol. 11, Issue 2
DOI: 10.1137/0111030

Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case
journal, January 2016

Garmanjani, R.; Júdice, D.; Vicente, L. N.
SIAM Journal on Optimization, Vol. 26, Issue 4
DOI: 10.1137/151005683

A derivative-free algorithm for linearly constrained optimization problems
journal, October 2013

Gumma, E. A. E.; Hashim, M. H. A.; Ali, M. Montaz
Computational Optimization and Applications, Vol. 57, Issue 3
DOI: 10.1007/s10589-013-9607-y

Computing a Trust Region Step
journal, September 1983

Moré, Jorge J.; Sorensen, D. C.
SIAM Journal on Scientific and Statistical Computing, Vol. 4, Issue 3
DOI: 10.1137/0904038

MILP models for the selection of a small set of well-distributed points
journal, January 2017

D'Ambrosio, Claudia; Nannicini, Giacomo; Sartor, Giorgio
Operations Research Letters, Vol. 45, Issue 1
DOI: 10.1016/j.orl.2016.11.004

Lipschitzian optimization without the Lipschitz constant
journal, October 1993

Jones, D. R.; Perttunen, C. D.; Stuckman, B. E.
Journal of Optimization Theory and Applications, Vol. 79, Issue 1
DOI: 10.1007/BF00941892

`` Direct Search'' Solution of Numerical and Statistical Problems
journal, April 1961

Hooke, Robert; Jeeves, T. A.
Journal of the ACM, Vol. 8, Issue 2
DOI: 10.1145/321062.321069

A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds
journal, April 1991

Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe
SIAM Journal on Numerical Analysis, Vol. 28, Issue 2
DOI: 10.1137/0728030

A Trust-Region Algorithm for Bi-Objective Stochastic Optimization
journal, January 2011

Kim, Sujin; Ryu, Jong-hyun
Procedia Computer Science, Vol. 4
DOI: 10.1016/j.procs.2011.04.153

Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations
journal, May 2015

Duchi, John C.; Jordan, Michael I.; Wainwright, Martin J.
IEEE Transactions on Information Theory, Vol. 61, Issue 5
DOI: 10.1109/TIT.2015.2409256

On the convergence of trust region algorithms for unconstrained minimization without derivatives
journal, April 2012

Powell, M. J. D.
Computational Optimization and Applications, Vol. 53, Issue 2
DOI: 10.1007/s10589-012-9483-x

Introductory Lectures on Convex Optimization
book, January 2004

Nesterov, Yurii
Applied Optimization
DOI: 10.1007/978-1-4419-8853-9

Pattern search methods for finite minimax problems
journal, March 2009

Ma, Junmei; Zhang, Xuesheng
Journal of Applied Mathematics and Computing, Vol. 32, Issue 2
DOI: 10.1007/s12190-009-0266-1

Worst case complexity of direct search
journal, December 2012

Vicente, L. N.
EURO Journal on Computational Optimization, Vol. 1, Issue 1-2
DOI: 10.1007/s13675-012-0003-7