Derivative-free optimization methods

Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M.

doi:10.1017/S0962492919000060

Title: Derivative-free optimization methods

Journal Article · Wed May 01 00:00:00 EDT 2019 · Acta Numerica

DOI:https://doi.org/10.1017/S0962492919000060· OSTI ID:1545343

Larson, Jeffrey ^[1]; Menickelly, Matt ^[1]; Wild, Stefan M. ^[1]

Argonne National Lab. (ANL), Argonne, IL (United States)

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 1545343

Journal Information:: Acta Numerica, Vol. 28, Issue 2010; ISSN 0962-4929

Publisher:: Cambridge University PressCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 139 works

Citation information provided by
Web of Science

References (281)

Sequential Design and Analysis of High-Accuracy and Low-Accuracy Computer Codes Xiong, Shifeng; Qian, Peter Z. G.; Wu, C. F. Jeff Technometrics, Vol. 55, Issue 1 https://doi.org/10.1080/00401706.2012.723572	journal	August 2012
Manifold Sampling for $\ell_1$ Nonconvex Optimization Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/15M1042097	journal	January 2016
Mesh adaptive direct search algorithms for mixed variable optimization Abramson, Mark A.; Audet, Charles; Chrissis, James W. Optimization Letters, Vol. 3, Issue 1 https://doi.org/10.1007/s11590-008-0089-2	journal	June 2008
A trust-region derivative-free algorithm for constrained optimization Conejo, P. D.; Karas, E. W.; Pedroso, L. G. Optimization Methods and Software, Vol. 30, Issue 6 https://doi.org/10.1080/10556788.2015.1026968	journal	April 2015
Convergence estimates for iterative minimization methods Karmanov, V. G. USSR Computational Mathematics and Mathematical Physics, Vol. 14, Issue 1 https://doi.org/10.1016/0041-5553(74)90131-1	journal	January 1974
Derivative-Free and Blackbox Optimization Audet, Charles; Hare, Warren Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/978-3-319-68913-5	book	January 2017
Algorithm 897: VTDIRECT95: Serial and parallel codes for the global optimization algorithm direct He, Jian; Watson, Layne T.; Sosonkina, Masha ACM Transactions on Mathematical Software, Vol. 36, Issue 3 https://doi.org/10.1145/1527286.1527291	journal	July 2009
A Pattern Search Filter Method for Nonlinear Programming without Derivatives Audet, Charles; Dennis, J. E. SIAM Journal on Optimization, Vol. 14, Issue 4 https://doi.org/10.1137/S105262340138983X	journal	January 2004
On the Local Convergence of Pattern Search Dolan, Elizabeth D.; Lewis, Robert Michael; Torczon, Virginia SIAM Journal on Optimization, Vol. 14, Issue 2 https://doi.org/10.1137/S1052623400374495	journal	January 2003
Global Convergence of Radial Basis Function Trust-Region Algorithms for Derivative-Free Optimization Wild, Stefan M.; Shoemaker, Christine SIAM Review, Vol. 55, Issue 2 https://doi.org/10.1137/120902434	journal	January 2013
An active-set trust-region method for derivative-free nonlinear bound-constrained optimization Gratton, Serge; Toint, Philippe L.; Tröltzsch, Anke Optimization Methods and Software, Vol. 26, Issue 4-5 https://doi.org/10.1080/10556788.2010.549231	journal	October 2011
Multiobjective Optimization Through a Series of Single-Objective Formulations Audet, Charles; Savard, Gilles; Zghal, Walid SIAM Journal on Optimization, Vol. 19, Issue 1 https://doi.org/10.1137/060677513	journal	January 2008
A Merit Function Approach for Direct Search Gratton, S.; Vicente, L. N. SIAM Journal on Optimization, Vol. 24, Issue 4 https://doi.org/10.1137/130917661	journal	January 2014
A View of Unconstrained Minimization Algorithms that Do Not Require Derivatives Powell, M. J. D. ACM Transactions on Mathematical Software, Vol. 1, Issue 2 https://doi.org/10.1145/355637.355638	journal	June 1975
Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers Kleinman, Nathan L.; Spall, James C.; Naiman, Daniel Q. Management Science, Vol. 45, Issue 11 https://doi.org/10.1287/mnsc.45.11.1570	journal	November 1999
Self-Correcting Geometry in Model-Based Algorithms for Derivative-Free Unconstrained Optimization Scheinberg, K.; Toint, Ph. L. SIAM Journal on Optimization, Vol. 20, Issue 6 https://doi.org/10.1137/090748536	journal	January 2010
A Derivative-Free Algorithm for Linearly Constrained Finite Minimax Problems Liuzzi, G.; Lucidi, S.; Sciandrone, M. SIAM Journal on Optimization, Vol. 16, Issue 4 https://doi.org/10.1137/040615821	journal	January 2006
Robust optimization of noisy blackbox problems using the Mesh Adaptive Direct Search algorithm Audet, Charles; Ihaddadene, Amina; Le Digabel, Sébastien Optimization Letters, Vol. 12, Issue 4 https://doi.org/10.1007/s11590-017-1226-6	journal	January 2018
Some aspects of the sequential design of experiments Robbins, Herbert Bulletin of the American Mathematical Society, Vol. 58, Issue 5 https://doi.org/10.1090/S0002-9904-1952-09620-8	journal	January 1952
On the local convergence of a derivative-free algorithm for least-squares minimization Zhang, Hongchao; Conn, Andrew R. Computational Optimization and Applications, Vol. 51, Issue 2 https://doi.org/10.1007/s10589-010-9367-x	journal	October 2010
A trust-region-based derivative free algorithm for mixed integer programming Newby, Eric; Ali, M. M. Computational Optimization and Applications, Vol. 60, Issue 1 https://doi.org/10.1007/s10589-014-9660-1	journal	April 2014
Numerical Analysis of $\mathcal{V}\mathcal{U}$-Decomposition, $\mathcal{U}$-Gradient, and $\mathcal{U}$-Hessian Approximations Hare, W. SIAM Journal on Optimization, Vol. 24, Issue 4 https://doi.org/10.1137/130933691	journal	January 2014
On Choosing Parameters in Retrospective-Approximation Algorithms for Stochastic Root Finding and Simulation Optimization Pasupathy, Raghu Operations Research, Vol. 58, Issue 4-part-1 https://doi.org/10.1287/opre.1090.0773	journal	August 2010
Random Gradient-Free Minimization of Convex Functions Nesterov, Yurii; Spokoiny, Vladimir Foundations of Computational Mathematics, Vol. 17, Issue 2 https://doi.org/10.1007/s10208-015-9296-2	journal	November 2015
A grid algorithm for bound constrained optimization of noisy functions Elster, Clemens; Neumaier, Arnold IMA Journal of Numerical Analysis, Vol. 15, Issue 4 https://doi.org/10.1093/imanum/15.4.585	journal	January 1995
Efficient calculation of regular simplex gradients Coope, Ian; Tappenden, Rachael Computational Optimization and Applications, Vol. 72, Issue 3 https://doi.org/10.1007/s10589-019-00063-3	journal	January 2019
Superlinear Convergence and Implicit Filtering Choi, T. D.; Kelley, C. T. SIAM Journal on Optimization, Vol. 10, Issue 4 https://doi.org/10.1137/S1052623499354096	journal	January 2000
Estimating Computational Noise Moré, Jorge J.; Wild, Stefan M. SIAM Journal on Scientific Computing, Vol. 33, Issue 3 https://doi.org/10.1137/100786125	journal	January 2011
A projected derivative-free algorithm for nonlinear equations with convex constraints La Cruz, William Optimization Methods and Software, Vol. 29, Issue 1 https://doi.org/10.1080/10556788.2012.721129	journal	July 2012
Sequential Search: A Method for Solving Constrained Optimization Problems Glass, H.; Cooper, L. Journal of the ACM, Vol. 12, Issue 1 https://doi.org/10.1145/321250.321256	journal	January 1965
Asymptotically efficient adaptive allocation rules Lai, T. L.; Robbins, Herbert Advances in Applied Mathematics, Vol. 6, Issue 1 https://doi.org/10.1016/0196-8858(85)90002-8	journal	March 1985
Worst case complexity of direct search under convexity Dodangeh, M.; Vicente, L. N. Mathematical Programming, Vol. 155, Issue 1-2 https://doi.org/10.1007/s10107-014-0847-0	journal	November 2014
Optimization with hidden constraints and embedded Monte Carlo computations Chen, Xiaojun; Kelley, C. T. Optimization and Engineering, Vol. 17, Issue 1 https://doi.org/10.1007/s11081-015-9302-1	journal	December 2015
Multidimensional Stochastic Approximation Methods Blum, Julius R. The Annals of Mathematical Statistics, Vol. 25, Issue 4 https://doi.org/10.1214/aoms/1177728659	journal	December 1954
A Nonderivative Version of the Gradient Sampling Algorithm for Nonsmooth Nonconvex Optimization Kiwiel, Krzysztof C. SIAM Journal on Optimization, Vol. 20, Issue 4 https://doi.org/10.1137/090748408	journal	January 2010
Pattern Search Algorithms for Bound Constrained Minimization Lewis, Robert Michael; Torczon, Virginia SIAM Journal on Optimization, Vol. 9, Issue 4 https://doi.org/10.1137/S1052623496300507	journal	January 1999
Cokriging-Based Sequential Design Strategies Using Fast Cross-Validation Techniques for Multi-Fidelity Computer Codes Le Gratiet, Loic; Cannamela, Claire Technometrics, Vol. 57, Issue 3 https://doi.org/10.1080/00401706.2014.928233	journal	July 2015
Kernel-based methods for Bandit convex optimization Bubeck, Sébastien; Lee, Yin Tat; Eldan, Ronen Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing - STOC 2017 https://doi.org/10.1145/3055399.3055403	conference	January 2017
Direct search algorithm for bilevel programming problems Mersha, Ayalew Getachew; Dempe, Stephan Computational Optimization and Applications, Vol. 49, Issue 1 https://doi.org/10.1007/s10589-009-9295-9	journal	September 2009
A method for the solution of certain non-linear problems in least squares Levenberg, Kenneth Quarterly of Applied Mathematics, Vol. 2, Issue 2 https://doi.org/10.1090/qam/10666	journal	January 1944
Spectral residual method without gradient information for solving large-scale nonlinear systems of equations La Cruz, William; Martínez, José Mario; Raydan, Marcos Mathematics of Computation, Vol. 75, Issue 255 https://doi.org/10.1090/S0025-5718-06-01840-0	journal	April 2006
The solution of variational and boundary value problems by the method of local variations Banichuk, N. V.; Petrov, V. M.; Chernous'ko, F. L. USSR Computational Mathematics and Mathematical Physics, Vol. 6, Issue 6 https://doi.org/10.1016/0041-5553(66)90158-3	journal	January 1966
Algorithm 856: APPSPACK 4.0: asynchronous parallel pattern search for derivative-free optimization Gray, Genetha A.; Kolda, Tamara G. ACM Transactions on Mathematical Software, Vol. 32, Issue 3 https://doi.org/10.1145/1163641.1163647	journal	September 2006
Compositions of convex functions and fully linear models Hare, W. Optimization Letters, Vol. 11, Issue 7 https://doi.org/10.1007/s11590-017-1117-x	journal	February 2017
Global Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms Wild, Stefan M.; Shoemaker, Christine SIAM Journal on Optimization, Vol. 21, Issue 3 https://doi.org/10.1137/09074927X	journal	July 2011
Sample mean based index policies by O (log n ) regret for the multi-armed bandit problem Agrawal, Rajeev Advances in Applied Probability, Vol. 27, Issue 4 https://doi.org/10.2307/1427934	journal	December 1995
Variable-Number Sample-Path Optimization Deng, Geng; Ferris, Michael C. Mathematical Programming, Vol. 117, Issue 1-2 https://doi.org/10.1007/s10107-007-0164-y	journal	July 2007
A Derivative-Free Trust-Region Method for Biobjective Optimization Ryu, Jong-hyun; Kim, Sujin SIAM Journal on Optimization, Vol. 24, Issue 1 https://doi.org/10.1137/120864738	journal	January 2014
Conditions for convergence of trust region algorithms for nonsmooth optimization Yuan, Y. Mathematical Programming, Vol. 31, Issue 2 https://doi.org/10.1007/BF02591750	journal	June 1985
Convergence Results for Generalized Pattern Search Algorithms are Tight Audet, Charles Optimization and Engineering, Vol. 5, Issue 2 https://doi.org/10.1023/B:OPTE.0000033370.66768.a9	journal	June 2004
Constrained derivative-free optimization on thin domains Martínez, J. M.; Sobral, F. N. C. Journal of Global Optimization, Vol. 56, Issue 3 https://doi.org/10.1007/s10898-012-9944-x	journal	July 2012
Direct search based on probabilistic feasible descent for bound and linearly constrained problems Gratton, S.; Royer, C. W.; Vicente, L. N. Computational Optimization and Applications, Vol. 72, Issue 3 https://doi.org/10.1007/s10589-019-00062-4	journal	January 2019
Stochastic online optimization. Single-point and multi-point non-linear multi-armed bandits. Convex and strongly-convex case Gasnikov, A. V.; Krymova, E. A.; Lagunovskaya, A. A. Automation and Remote Control, Vol. 78, Issue 2 https://doi.org/10.1134/S0005117917020035	journal	February 2017
A Derivative-Free Algorithm for Inequality Constrained Nonlinear Programming via Smoothing of an $\ell_\infty$ Penalty Function Liuzzi, G.; Lucidi, S. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/070711451	journal	January 2009
Design and implementation of a massively parallel version of DIRECT He, Jian; Verstak, Alex; Watson, Layne T. Computational Optimization and Applications, Vol. 40, Issue 2 https://doi.org/10.1007/s10589-007-9092-2	journal	October 2007
Approximation Methods which Converge with Probability one Blum, Julius R. The Annals of Mathematical Statistics, Vol. 25, Issue 2 https://doi.org/10.1214/aoms/1177728794	journal	June 1954
Derivative-Free Methods for Mixed-Integer Constrained Optimization Problems Liuzzi, Giampaolo; Lucidi, Stefano; Rinaldi, Francesco Journal of Optimization Theory and Applications, Vol. 164, Issue 3 https://doi.org/10.1007/s10957-014-0617-4	journal	July 2014
MISO: mixed-integer surrogate optimization framework Müller, Juliane Optimization and Engineering, Vol. 17, Issue 1 https://doi.org/10.1007/s11081-015-9281-2	journal	June 2015
Asynchronous Parallel Pattern Search for Nonlinear Optimization Hough, Patricia D.; Kolda, Tamara G.; Torczon, Virginia J. SIAM Journal on Scientific Computing, Vol. 23, Issue 1 https://doi.org/10.1137/S1064827599365823	journal	January 2001
On the convergence of a wide range of trust region methods for unconstrained optimization Powell, M. J. D. IMA Journal of Numerical Analysis, Vol. 30, Issue 1 https://doi.org/10.1093/imanum/drp021	journal	September 2009
Parallel Space Decomposition of the Mesh Adaptive Direct Search Algorithm Audet, Charles; Dennis, J. E.; Digabel, Sébastien Le SIAM Journal on Optimization, Vol. 19, Issue 3 https://doi.org/10.1137/070707518	journal	January 2008
Incorporating minimum Frobenius norm models in direct search Custódio, A. L.; Rocha, H.; Vicente, L. N. Computational Optimization and Applications, Vol. 46, Issue 2 https://doi.org/10.1007/s10589-009-9283-0	journal	August 2009
A batch, derivative-free algorithm for finding multiple local minima Larson, Jeffrey; Wild, Stefan M. Optimization and Engineering, Vol. 17, Issue 1 https://doi.org/10.1007/s11081-015-9289-7	journal	October 2015
Generalized pattern searches with derivative information Abramson, Mark A.; Audet, Charles; Dennis, J. E. Mathematical Programming, Vol. 100, Issue 1 https://doi.org/10.1007/s10107-003-0484-5	journal	November 2003
A derivative-free trust-region algorithm for composite nonsmooth optimization Grapiglia, Geovani Nunes; Yuan, Jinyun; Yuan, Ya-xiang Computational and Applied Mathematics, Vol. 35, Issue 2 https://doi.org/10.1007/s40314-014-0201-4	journal	December 2014
Pattern search ranking and selection algorithms for mixed variable simulation-based optimization Sriver, Todd A.; Chrissis, James W.; Abramson, Mark A. European Journal of Operational Research, Vol. 198, Issue 3 https://doi.org/10.1016/j.ejor.2008.10.020	journal	November 2009
Derivative-free optimization: a review of algorithms and comparison of software implementations Rios, Luis Miguel; Sahinidis, Nikolaos V. Journal of Global Optimization, Vol. 56, Issue 3 https://doi.org/10.1007/s10898-012-9951-y	journal	July 2012
Mesh Adaptive Direct Search Algorithms for Constrained Optimization Audet, Charles; Dennis, J. E. SIAM Journal on Optimization, Vol. 17, Issue 1 https://doi.org/10.1137/040603371	journal	January 2006
A trust-region framework for managing the use of approximation models in optimization Alexandrov, N. M.; Dennis, J. E.; Lewis, R. M. Structural Optimization, Vol. 15, Issue 1 https://doi.org/10.1007/BF01197433	journal	February 1998
On the Oracle Complexity of First-Order and Derivative-Free Algorithms for Smooth Nonconvex Minimization Cartis, Coralia; Gould, Nicholas I. M.; Toint, Philippe L. SIAM Journal on Optimization, Vol. 22, Issue 1 https://doi.org/10.1137/100812276	journal	January 2012
A stochastic method for global optimization Boender, C. G. E.; Rinnooy Kan, A. H. G.; Timmer, G. T. Mathematical Programming, Vol. 22, Issue 1 https://doi.org/10.1007/BF01581033	journal	December 1982
A class of derivative-free methods for large-scale nonlinear monotone equations Li, Q.; Li, D. -H. IMA Journal of Numerical Analysis, Vol. 31, Issue 4 https://doi.org/10.1093/imanum/drq015	journal	January 2011
A Convergent Variant of the Nelder–Mead Algorithm Price, C. J.; Coope, I. D.; Byatt, D. Journal of Optimization Theory and Applications, Vol. 113, Issue 1 https://doi.org/10.1023/A:1014849028575	journal	April 2002
VXQR: derivative-free unconstrained optimization based on QR factorizations Neumaier, Arnold; Fendl, Hannes; Schilly, Harald Soft Computing, Vol. 15, Issue 11 https://doi.org/10.1007/s00500-010-0652-5	journal	September 2010
Geometry of interpolation sets in derivative free optimization Conn, A. R.; Scheinberg, K.; Vicente, Luís N. Mathematical Programming, Vol. 111, Issue 1-2 https://doi.org/10.1007/s10107-006-0073-5	journal	December 2006
On the Number of Iterations of Piyavskii's Global Optimization Algorithm Hansen, P.; Jaumard, B.; Lu, S. -H. Mathematics of Operations Research, Vol. 16, Issue 2 https://doi.org/10.1287/moor.16.2.334	journal	May 1991
A derivative-free line search and global convergence of Broyden-like method for nonlinear equations Li, Dong-Hui; Fukushima, Masao Optimization Methods and Software, Vol. 13, Issue 3 https://doi.org/10.1080/10556780008805782	journal	January 2000
Beyond symmetric Broyden for updating quadratic models in minimization without derivatives Powell, M. J. D. Mathematical Programming, Vol. 138, Issue 1-2 https://doi.org/10.1007/s10107-011-0510-y	journal	January 2012
Performance Modeling and Analysis of a Massively Parallel Direct—Part 1 The International Journal of High Performance Computing Applications, Vol. 23, Issue 1 https://doi.org/10.1177/1094342008098462	journal	February 2009
A subclass of generating set search with convergence to second-order stationary points Abramson, Mark A.; Frimannslund, Lennart; Steihaug, Trond Optimization Methods and Software, Vol. 29, Issue 5 https://doi.org/10.1080/10556788.2013.808198	journal	June 2013
Performance Modeling and Analysis of a Massively Parallel Direct—Part 2 The International Journal of High Performance Computing Applications, Vol. 23, Issue 1 https://doi.org/10.1177/1094342008098463	journal	February 2009
Direct search methods: then and now Lewis, Robert Michael; Torczon, Virginia; Trosset, Michael W. Journal of Computational and Applied Mathematics, Vol. 124, Issue 1-2 https://doi.org/10.1016/S0377-0427(00)00423-4	journal	December 2000
OrthoMADS: A Deterministic MADS Instance with Orthogonal Directions Abramson, Mark A.; Audet, Charles; Dennis,, J. E. SIAM Journal on Optimization, Vol. 20, Issue 2 https://doi.org/10.1137/080716980	journal	January 2009
Constrained optimization involving expensive function evaluations: A sequential approach Brekelmans, Ruud; Driessen, Lonneke; Hamers, Herbert European Journal of Operational Research, Vol. 160, Issue 1 https://doi.org/10.1016/j.ejor.2003.10.009	journal	January 2005
Boosters: A Derivative-Free Algorithm Based on Radial Basis Functions Oeuvray, R.; Bierlaire, M. International Journal of Modelling and Simulation, Vol. 29, Issue 1 https://doi.org/10.1080/02286203.2009.11442507	journal	January 2009
Two-Point Step Size Gradient Methods Barzilai, Jonathan; Borwein, Jonathan M. IMA Journal of Numerical Analysis, Vol. 8, Issue 1 https://doi.org/10.1093/imanum/8.1.141	journal	January 1988
Stochastic Nelder–Mead simplex method – A new globally convergent direct search method for simulation optimization Chang, Kuo-Hao European Journal of Operational Research, Vol. 220, Issue 3 https://doi.org/10.1016/j.ejor.2012.02.028	journal	August 2012
Stationarity Results for Generating Set Search for Linearly Constrained Optimization Kolda, Tamara G.; Lewis, Robert Michael; Torczon, Virginia SIAM Journal on Optimization, Vol. 17, Issue 4 https://doi.org/10.1137/S1052623403433638	journal	January 2007
An Implicit Filtering Algorithm for Optimization of Functions with Many Local Minima Gilmore, P.; Kelley, C. T. SIAM Journal on Optimization, Vol. 5, Issue 2 https://doi.org/10.1137/0805015	journal	May 1995
Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods Kolda, Tamara G.; Lewis, Robert Michael; Torczon, Virginia SIAM Review, Vol. 45, Issue 3 https://doi.org/10.1137/S003614450242889	journal	January 2003
Inexact Restoration Method for Derivative-Free Optimization with Smooth Constraints Bueno, L. F.; Friedlander, A.; Martínez, J. M. SIAM Journal on Optimization, Vol. 23, Issue 2 https://doi.org/10.1137/110856253	journal	January 2013
A Derivative-Free Approach to Constrained Multiobjective Nonsmooth Optimization Liuzzi, G.; Lucidi, S.; Rinaldi, F. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/15M1037810	journal	January 2016
Global convergence and stabilization of unconstrained minimization methods without derivatives Grippo, L.; Lampariello, F.; Lucidi, S. Journal of Optimization Theory and Applications, Vol. 56, Issue 3 https://doi.org/10.1007/BF00939550	journal	March 1988
A derivative-free comirror algorithm for convex optimization Bauschke, Heinz H.; Hare, Warren L.; Moursi, Walaa M. Optimization Methods and Software, Vol. 30, Issue 4 https://doi.org/10.1080/10556788.2014.968158	journal	October 2014
An Algorithm for Least-Squares Estimation of Nonlinear Parameters Marquardt, Donald W. Journal of the Society for Industrial and Applied Mathematics, Vol. 11, Issue 2 https://doi.org/10.1137/0111030	journal	June 1963
Trust-Region Methods Without Using Derivatives: Worst Case Complexity and the NonSmooth Case Garmanjani, R.; Júdice, D.; Vicente, L. N. SIAM Journal on Optimization, Vol. 26, Issue 4 https://doi.org/10.1137/151005683	journal	January 2016
A derivative-free algorithm for linearly constrained optimization problems Gumma, E. A. E.; Hashim, M. H. A.; Ali, M. Montaz Computational Optimization and Applications, Vol. 57, Issue 3 https://doi.org/10.1007/s10589-013-9607-y	journal	October 2013
Computing a Trust Region Step Moré, Jorge J.; Sorensen, D. C. SIAM Journal on Scientific and Statistical Computing, Vol. 4, Issue 3 https://doi.org/10.1137/0904038	journal	September 1983
MILP models for the selection of a small set of well-distributed points D'Ambrosio, Claudia; Nannicini, Giacomo; Sartor, Giorgio Operations Research Letters, Vol. 45, Issue 1 https://doi.org/10.1016/j.orl.2016.11.004	journal	January 2017
Lipschitzian optimization without the Lipschitz constant Jones, D. R.; Perttunen, C. D.; Stuckman, B. E. Journal of Optimization Theory and Applications, Vol. 79, Issue 1 https://doi.org/10.1007/BF00941892	journal	October 1993
`` Direct Search'' Solution of Numerical and Statistical Problems Hooke, Robert; Jeeves, T. A. Journal of the ACM, Vol. 8, Issue 2 https://doi.org/10.1145/321062.321069	journal	April 1961
A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe SIAM Journal on Numerical Analysis, Vol. 28, Issue 2 https://doi.org/10.1137/0728030	journal	April 1991
A Trust-Region Algorithm for Bi-Objective Stochastic Optimization Kim, Sujin; Ryu, Jong-hyun Procedia Computer Science, Vol. 4 https://doi.org/10.1016/j.procs.2011.04.153	journal	January 2011
Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations Duchi, John C.; Jordan, Michael I.; Wainwright, Martin J. IEEE Transactions on Information Theory, Vol. 61, Issue 5 https://doi.org/10.1109/TIT.2015.2409256	journal	May 2015
On the convergence of trust region algorithms for unconstrained minimization without derivatives Powell, M. J. D. Computational Optimization and Applications, Vol. 53, Issue 2 https://doi.org/10.1007/s10589-012-9483-x	journal	April 2012
Introductory Lectures on Convex Optimization Nesterov, Yurii Applied Optimization https://doi.org/10.1007/978-1-4419-8853-9	book	January 2004
Pattern search methods for finite minimax problems Ma, Junmei; Zhang, Xuesheng Journal of Applied Mathematics and Computing, Vol. 32, Issue 2 https://doi.org/10.1007/s12190-009-0266-1	journal	March 2009
Worst case complexity of direct search Vicente, L. N. EURO Journal on Computational Optimization, Vol. 1, Issue 1-2 https://doi.org/10.1007/s13675-012-0003-7	journal	December 2012
Derivative-free optimization methods for finite minimax problems Hare, Warren; Macklem, Mason Optimization Methods and Software, Vol. 28, Issue 2 https://doi.org/10.1080/10556788.2011.638923	journal	April 2013
On sequential and parallel non-monotone derivative-free algorithms for box constrained optimization García-Palomares, Ubaldo M.; García-Urrea, Ildemaro J.; Rodríguez-Hernández, Pedro S. Optimization Methods and Software, Vol. 28, Issue 6 https://doi.org/10.1080/10556788.2012.693926	journal	June 2012
Global convergence of a derivative-free inexact restoration filter algorithm for nonlinear programming Ferreira, P. S.; Karas, E. W.; Sachine, M. Optimization, Vol. 66, Issue 2 https://doi.org/10.1080/02331934.2016.1263629	journal	November 2016
Optimizing partially separable functions without derivatives Colson, Benoît; Toint, Philippe L. Optimization Methods and Software, Vol. 20, Issue 4-5 https://doi.org/10.1080/10556780500140227	journal	August 2005
Convergence of Trust-Region Methods Based on Probabilistic Models Bandeira, A. S.; Scheinberg, K.; Vicente, L. N. SIAM Journal on Optimization, Vol. 24, Issue 3 https://doi.org/10.1137/130915984	journal	January 2014
CONDOR, a new parallel, constrained extension of Powell's UOBYQA algorithm: Experimental results and comparison with the DFO algorithm Vanden Berghen, Frank; Bersini, Hugues Journal of Computational and Applied Mathematics, Vol. 181, Issue 1 https://doi.org/10.1016/j.cam.2004.11.029	journal	September 2005
Asynchronously parallel optimization solver for finding multiple minima Larson, Jeffrey; Wild, Stefan M. Mathematical Programming Computation, Vol. 10, Issue 3 https://doi.org/10.1007/s12532-017-0131-4	journal	February 2018
Provably Convergent Multifidelity Optimization Algorithm Not Requiring High-Fidelity Derivatives March, Andrew; Willcox, Karen AIAA Journal, Vol. 50, Issue 5 https://doi.org/10.2514/1.J051125	journal	May 2012
On the convergence of the UOBYQA method Han, Lixing; Liu, Guanghui Journal of Applied Mathematics and Computing, Vol. 16, Issue 1-2 https://doi.org/10.1007/BF02936156	journal	March 2004
Additive Scaling and the DIRECT Algorithm Finkel, D. E.; Kelley, C. T. Journal of Global Optimization, Vol. 36, Issue 4 https://doi.org/10.1007/s10898-006-9029-9	journal	June 2006
Bayesian Guided Pattern Search for Robust Local Optimization Taddy, Matthew A.; Lee, Herbert K. H.; Gray, Genetha A. Technometrics, Vol. 51, Issue 4 https://doi.org/10.1198/TECH.2009.08007	journal	November 2009
Dud, A Derivative-Free Algorithm for Nonlinear Least Squares Ralston, Mary L.; Jennrich, Robert I. Technometrics, Vol. 20, Issue 1 https://doi.org/10.1080/00401706.1978.10489610	journal	February 1978
An implicit filtering algorithm for derivative-free multiobjective optimization with box constraints Cocchi, G.; Liuzzi, G.; Papini, A. Computational Optimization and Applications, Vol. 69, Issue 2 https://doi.org/10.1007/s10589-017-9953-2	journal	October 2017
Sequential Application of Simplex Designs in Optimisation and Evolutionary Operation Spendley, W.; Hext, G. R.; Himsworth, F. R. Technometrics, Vol. 4, Issue 4 https://doi.org/10.1080/00401706.1962.10490033	journal	November 1962
Stochastic derivative-free optimization using a trust region framework Larson, Jeffrey; Billups, Stephen C. Computational Optimization and Applications, Vol. 64, Issue 3 https://doi.org/10.1007/s10589-016-9827-z	journal	February 2016
On the optimal order of worst case complexity of direct search Dodangeh, M.; Vicente, L. N.; Zhang, Z. Optimization Letters, Vol. 10, Issue 4 https://doi.org/10.1007/s11590-015-0908-1	journal	June 2015
Stopping criteria for linesearch methods without derivatives De Leone, Renato; Gaudioso, Manlio; Grippo, Luigi Mathematical Programming, Vol. 30, Issue 3 https://doi.org/10.1007/BF02591934	journal	October 1984
Derivative-Free Optimization of Expensive Functions with Computational Error Using Weighted Regression Billups, Stephen C.; Larson, Jeffrey; Graf, Peter SIAM Journal on Optimization, Vol. 23, Issue 1 https://doi.org/10.1137/100814688	journal	January 2013
Using Sampling and Simplex Derivatives in Pattern Search Methods Custódio, A. L.; Vicente, L. N. SIAM Journal on Optimization, Vol. 18, Issue 2 https://doi.org/10.1137/050646706	journal	January 2007
Solving Nonlinear Programs Without Using Analytic Derivatives May, Jerrold H. Operations Research, Vol. 27, Issue 3 https://doi.org/10.1287/opre.27.3.457	journal	June 1979
A second-order globally convergent direct-search method and its worst-case complexity Gratton, S.; Royer, C. W.; Vicente, L. N. Optimization, Vol. 65, Issue 6 https://doi.org/10.1080/02331934.2015.1124271	journal	October 2015
The Nonstochastic Multiarmed Bandit Problem Auer, Peter; Cesa-Bianchi, Nicolò; Freund, Yoav SIAM Journal on Computing, Vol. 32, Issue 1 https://doi.org/10.1137/S0097539701398375	journal	January 2002
Robust optimization with simulated annealing Bertsimas, Dimitris; Nohadani, Omid Journal of Global Optimization, Vol. 48, Issue 2 https://doi.org/10.1007/s10898-009-9496-x	journal	December 2009
A superlinearly convergent algorithm for minimization without evaluating derivatives Mifflin, Robert Mathematical Programming, Vol. 9, Issue 1 https://doi.org/10.1007/BF01681333	journal	December 1975
SNOBFIT -- Stable Noisy Optimization by Branch and Fit Huyer, Waltraud; Neumaier, Arnold ACM Transactions on Mathematical Software, Vol. 35, Issue 2 https://doi.org/10.1145/1377612.1377613	journal	July 2008
Pure adaptive search in global optimization Zabinsky, Zelda B.; Smith, Robert L. Mathematical Programming, Vol. 53, Issue 1-3 https://doi.org/10.1007/BF01585710	journal	January 1992
Fortified-Descent Simplicial Search Method: A General Approach Tseng, Paul SIAM Journal on Optimization, Vol. 10, Issue 1 https://doi.org/10.1137/S1052623495282857	journal	January 1999
Do you trust derivatives or differences? Moré, Jorge J.; Wild, Stefan M. Journal of Computational Physics, Vol. 273 https://doi.org/10.1016/j.jcp.2014.04.056	journal	September 2014
A derivative-free approximate gradient sampling algorithm for finite minimax problems Hare, W.; Nutini, J. Computational Optimization and Applications, Vol. 56, Issue 1 https://doi.org/10.1007/s10589-013-9547-6	journal	March 2013
Evaluating Derivatives Griewank, Andreas; Walther, Andrea https://doi.org/10.1137/1.9780898717761	book	January 2008
A sequential quadratic programming algorithm for equality-constrained optimization without derivatives Tröltzsch, Anke Optimization Letters, Vol. 10, Issue 2 https://doi.org/10.1007/s11590-014-0830-y	journal	November 2014
Stochastic optimization using a trust-region method and random models Chen, R.; Menickelly, M.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1141-8	journal	April 2017
Generalized Pattern Search methods for a class of nonsmooth optimization problems with structure Bogani, C.; Gasparo, M. G.; Papini, A. Journal of Computational and Applied Mathematics, Vol. 229, Issue 1 https://doi.org/10.1016/j.cam.2008.10.047	journal	July 2009
A Linesearch-Based Derivative-Free Approach for Nonsmooth Constrained Optimization Fasano, G.; Liuzzi, G.; Lucidi, S. SIAM Journal on Optimization, Vol. 24, Issue 3 https://doi.org/10.1137/130940037	journal	January 2014
GLODS: Global and Local Optimization using Direct Search Custódio, A. L.; Madeira, J. F. A. Journal of Global Optimization, Vol. 62, Issue 1 https://doi.org/10.1007/s10898-014-0224-9	journal	August 2014
Multivariate stochastic approximation using a simultaneous perturbation gradient approximation Spall, J. C. IEEE Transactions on Automatic Control, Vol. 37, Issue 3 https://doi.org/10.1109/9.119632	journal	March 1992
Second-Order Behavior of Pattern Search Abramson, Mark A. SIAM Journal on Optimization, Vol. 16, Issue 2 https://doi.org/10.1137/04060367X	journal	January 2005
The calculus of simplex gradients Regis, Rommel G. Optimization Letters, Vol. 9, Issue 5 https://doi.org/10.1007/s11590-014-0815-x	journal	October 2014
New horizons in sphere-packing theory, part II: lattice-based derivative-free optimization via global surrogates Belitz, Paul; Bewley, Thomas Journal of Global Optimization, Vol. 56, Issue 1 https://doi.org/10.1007/s10898-012-9866-7	journal	March 2012
Grid Restrained Nelder-Mead Algorithm Bűrmen, Árpád; Puhan, Janez; Tuma, Tadej Computational Optimization and Applications, Vol. 34, Issue 3 https://doi.org/10.1007/s10589-005-3912-z	journal	March 2006
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems Bubeck, Sébastien Foundations and Trends® in Machine Learning, Vol. 5, Issue 1 https://doi.org/10.1561/2200000024	journal	January 2012
The Simplex Gradient and Noisy Optimization Problems Bortz, D. M.; Kelley, C. T. Computational Methods for Optimal Design and Control https://doi.org/10.1007/978-1-4612-1780-0_5	book	January 1998
Bilevel derivative-free optimization and its application to robust optimization Conn, A. R.; Vicente, L. N. Optimization Methods and Software, Vol. 27, Issue 3 https://doi.org/10.1080/10556788.2010.547579	journal	June 2012
Computing Forward-Difference Intervals for Numerical Optimization Gill, Philip E.; Murray, Walter; Saunders, Michael A. SIAM Journal on Scientific and Statistical Computing, Vol. 4, Issue 2 https://doi.org/10.1137/0904025	journal	June 1983
On Lipschitz optimization based on gray-box piecewise linearization Griewank, Andreas; Walther, Andrea; Fiege, Sabrina Mathematical Programming, Vol. 158, Issue 1-2 https://doi.org/10.1007/s10107-015-0934-x	journal	July 2015
Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points Regis, Rommel G. Engineering Optimization, Vol. 46, Issue 2 https://doi.org/10.1080/0305215X.2013.765000	journal	February 2013
A mesh adaptive direct search algorithm for multiobjective optimization Audet, Charles; Savard, Gilles; Zghal, Walid European Journal of Operational Research, Vol. 204, Issue 3 https://doi.org/10.1016/j.ejor.2009.11.010	journal	August 2010
Direct Multisearch for Multiobjective Optimization Custódio, A. L.; Madeira, J. F. A.; Vaz, A. I. F. SIAM Journal on Optimization, Vol. 21, Issue 3 https://doi.org/10.1137/10079731X	journal	July 2011
Optimization of Convex Functions with Random Pursuit Stich, S. U.; Müller, C. L.; Gärtner, B. SIAM Journal on Optimization, Vol. 23, Issue 2 https://doi.org/10.1137/110853613	journal	January 2013
Exploiting Known Structures to Approximate Normal Cones Davis, Chad; Hare, Warren Mathematics of Operations Research, Vol. 38, Issue 4 https://doi.org/10.1287/moor.2013.0590	journal	November 2013
On fast trust region methods for quadratic models with linear constraints Powell, M. J. D. Mathematical Programming Computation, Vol. 7, Issue 3 https://doi.org/10.1007/s12532-015-0084-4	journal	June 2015
BFO, A Trainable Derivative-free Brute Force Optimizer for Nonlinear Bound-constrained Optimization and Equilibrium Computations with Continuous and Discrete Variables Porcelli, Margherita; Toint, Philippe L. ACM Transactions on Mathematical Software, Vol. 44, Issue 1 https://doi.org/10.1145/3085592	journal	June 2017
Piecewise Partially Separable Functions and a Derivative-free Algorithm for Large Scale Nonsmooth Optimization Bagirov, Adil M.; Ugon, Julien Journal of Global Optimization, Vol. 35, Issue 2 https://doi.org/10.1007/s10898-005-3834-4	journal	June 2006
Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical Points Conn, Andrew R.; Scheinberg, Katya; Vicente, Luís N. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/060673424	journal	January 2009
A Derivative-Free Algorithm for Least-Squares Minimization Zhang, Hongchao; Conn, Andrew R.; Scheinberg, Katya SIAM Journal on Optimization, Vol. 20, Issue 6 https://doi.org/10.1137/09075531X	journal	January 2010
An adaptive multiquadric radial basis function method for expensive black-box mixed-integer nonlinear constrained optimization Rashid, Kashif; Ambani, Saumil; Cetinkaya, Eren Engineering Optimization, Vol. 45, Issue 2 https://doi.org/10.1080/0305215X.2012.665450	journal	February 2013
A derivative-free nonmonotone line-search technique for unconstrained optimization Diniz-Ehrhardt, M. A.; Martínez, J. M.; Raydan, M. Journal of Computational and Applied Mathematics, Vol. 219, Issue 2 https://doi.org/10.1016/j.cam.2007.07.017	journal	October 2008
A method for stochastic constrained optimization using derivative-free surrogate pattern search and collocation Sankaran, Sethuraman; Audet, Charles; Marsden, Alison L. Journal of Computational Physics, Vol. 229, Issue 12 https://doi.org/10.1016/j.jcp.2010.03.005	journal	June 2010
New Sequential and Parallel Derivative-Free Algorithms for Unconstrained Minimization García-Palomares, U. M.; Rodríguez, J. F. SIAM Journal on Optimization, Vol. 13, Issue 1 https://doi.org/10.1137/S1052623400370606	journal	January 2002
On the Convergence of the Multidirectional Search Algorithm Torczon, Virginia SIAM Journal on Optimization, Vol. 1, Issue 1 https://doi.org/10.1137/0801010	journal	February 1991
Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation Brown, Kenneth M.; Dennis, J. E. Numerische Mathematik, Vol. 18, Issue 4 https://doi.org/10.1007/BF01404679	journal	August 1971
On Sampling Rates in Simulation-Based Recursions Pasupathy, Raghu; Glynn, Peter; Ghosh, Soumyadip SIAM Journal on Optimization, Vol. 28, Issue 1 https://doi.org/10.1137/140951679	journal	January 2018
ORBIT: Optimization by Radial Basis Function Interpolation in Trust-Regions Wild, Stefan M.; Regis, Rommel G.; Shoemaker, Christine A. SIAM Journal on Scientific Computing, Vol. 30, Issue 6 https://doi.org/10.1137/070691814	journal	January 2008
A Stochastic Approximation Method Robbins, Herbert; Monro, Sutton The Annals of Mathematical Statistics, Vol. 22, Issue 3 https://doi.org/10.1214/aoms/1177729586	journal	September 1951
Estimating Derivatives of Noisy Simulations Moré, Jorge J.; Wild, Stefan M. ACM Transactions on Mathematical Software, Vol. 38, Issue 3 https://doi.org/10.1145/2168773.2168777	journal	April 2012
Chapter 40: POUNDERS in TAO: Solving Derivative-Free Nonlinear Least-Squares Problems with POUNDERS Wild, Stefan M.; Terlaky, Tamás; Anjos, Miguel F. Advances and Trends in Optimization with Engineering Applications https://doi.org/10.1137/1.9781611974683.ch40	book	April 2017
On the construction of quadratic models for derivative-free trust-region algorithms Verdério, Adriano; Karas, Elizabeth W.; Pedroso, Lucas G. EURO Journal on Computational Optimization, Vol. 5, Issue 4 https://doi.org/10.1007/s13675-017-0081-7	journal	February 2017
Approximate norm descent methods for constrained nonlinear systems Morini, Benedetta; Porcelli, Margherita; Toint, Philippe L. Mathematics of Computation, Vol. 87, Issue 311 https://doi.org/10.1090/mcom/3251	journal	May 2017
SO-MI: A surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems Müller, Juliane; Shoemaker, Christine A.; Piché, Robert Computers & Operations Research, Vol. 40, Issue 5 https://doi.org/10.1016/j.cor.2012.08.022	journal	May 2013
An inexact restoration derivative-free filter method for nonlinear programming Echebest, N.; Schuverdt, M. L.; Vignau, R. P. Computational and Applied Mathematics, Vol. 36, Issue 1 https://doi.org/10.1007/s40314-015-0253-0	journal	July 2015
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
Taking the Human Out of the Loop: A Review of Bayesian Optimization Shahriari, Bobak; Swersky, Kevin; Wang, Ziyu Proceedings of the IEEE, Vol. 104, Issue 1 https://doi.org/10.1109/JPROC.2015.2494218	journal	January 2016
Precision Control for Generalized Pattern Search Algorithms with Adaptive Precision Function Evaluations Polak, Elijah; Wetter, Michael SIAM Journal on Optimization, Vol. 16, Issue 3 https://doi.org/10.1137/040605527	journal	January 2006
Analysis of direct searches for discontinuous functions Vicente, L. N.; Custódio, A. L. Mathematical Programming, Vol. 133, Issue 1-2 https://doi.org/10.1007/s10107-010-0429-8	journal	December 2010
Algorithm 909: NOMAD: Nonlinear Optimization with the MADS Algorithm Le Digabel, Sébastien ACM Transactions on Mathematical Software, Vol. 37, Issue 4 https://doi.org/10.1145/1916461.1916468	journal	February 2011
On the use of quadratic models in unconstrained minimization without derivatives Powell, M. J. D. Optimization Methods and Software, Vol. 19, Issue 3-4 https://doi.org/10.1080/10556780410001661450	journal	June 2004
Linear equalities in blackbox optimization Audet, Charles; Le Digabel, Sébastien; Peyrega, Mathilde Computational Optimization and Applications, Vol. 61, Issue 1 https://doi.org/10.1007/s10589-014-9708-2	journal	October 2014
Efficient Global Optimization of Expensive Black-Box Functions Jones, Donald R.; Schonlau, Matthias; Welch, William J. Journal of Global Optimization, Vol. 13, Issue 4, p. 455-492 https://doi.org/10.1023/A:1008306431147	journal	January 1998
Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming Ghadimi, Saeed; Lan, Guanghui SIAM Journal on Optimization, Vol. 23, Issue 4 https://doi.org/10.1137/120880811	journal	January 2013
A trust region-based two phase algorithm for constrained black-box and grey-box optimization with infeasible initial point Bajaj, Ishan; Iyer, Shachit S.; Faruque Hasan, M. M. Computers & Chemical Engineering, Vol. 116 https://doi.org/10.1016/j.compchemeng.2017.12.011	journal	August 2018
Finite-time Analysis of the Multiarmed Bandit Problem Auer, Peter; Cesa-Bianchi, Nicolò; Fischer, Paul Machine Learning, Vol. 47, Issue 2/3, p. 235-256 https://doi.org/10.1023/A:1013689704352	journal	May 2002
Iterative Methods for Optimization Kelley, C. T. https://doi.org/10.1137/1.9781611970920	book	January 1999
Direct Search Methods on Parallel Machines Dennis, Jr., J. E.; Torczon, Virginia SIAM Journal on Optimization, Vol. 1, Issue 4 https://doi.org/10.1137/0801027	journal	November 1991
Coordinate search algorithms in multilevel optimization Frandi, Emanuele; Papini, Alessandra Optimization Methods and Software, Vol. 29, Issue 5 https://doi.org/10.1080/10556788.2013.841691	journal	November 2013
Derivative-free methods for bound constrained mixed-integer optimization Liuzzi, G.; Lucidi, S.; Rinaldi, F. Computational Optimization and Applications, Vol. 53, Issue 2 https://doi.org/10.1007/s10589-011-9405-3	journal	April 2011
Surrogate-based methods for black-box optimization: Surrogate-based methods for black-box optimization Vu, Ky Khac; D'Ambrosio, Claudia; Hamadi, Youssef International Transactions in Operational Research, Vol. 24, Issue 3 https://doi.org/10.1111/itor.12292	journal	April 2016
A Stochastic Radial Basis Function Method for the Global Optimization of Expensive Functions Regis, Rommel G.; Shoemaker, Christine A. INFORMS Journal on Computing, Vol. 19, Issue 4 https://doi.org/10.1287/ijoc.1060.0182	journal	November 2007
An Application of Chung's Lemma to the Kiefer-Wolfowitz Stochastic Approximation Procedure Derman, Cyrus The Annals of Mathematical Statistics, Vol. 27, Issue 2 https://doi.org/10.1214/aoms/1177728277	journal	June 1956
Optimal Condenser Design by Geometric Programming Avriel, Mordecai; Wilde, D. J. Industrial & Engineering Chemistry Process Design and Development, Vol. 6, Issue 2 https://doi.org/10.1021/i260022a018	journal	April 1967
Nelder-Mead Simplex Modifications for Simulation Optimization Barton, Russell R.; Ivey, John S. Management Science, Vol. 42, Issue 7 https://doi.org/10.1287/mnsc.42.7.954	journal	July 1996
CONORBIT: constrained optimization by radial basis function interpolation in trust regions Regis, Rommel G.; Wild, Stefan M. Optimization Methods and Software, Vol. 32, Issue 3 https://doi.org/10.1080/10556788.2016.1226305	journal	August 2016
A multigrid approach to discretized optimization problems Nash, Stephen G. Optimization Methods and Software, Vol. 14, Issue 1-2 https://doi.org/10.1080/10556780008805795	journal	January 2000
Frame Based Methods for Unconstrained Optimization Coope, I. D.; Price, C. J. Journal of Optimization Theory and Applications, Vol. 107, Issue 2 https://doi.org/10.1023/A:1026429319405	journal	November 2000
Mesh adaptive direct search with second directional derivative-based Hessian update Bűrmen, Árpád; Olenšek, Jernej; Tuma, Tadej Computational Optimization and Applications, Vol. 62, Issue 3 https://doi.org/10.1007/s10589-015-9753-5	journal	April 2015
On the properties of positive spanning sets and positive bases Regis, Rommel G. Optimization and Engineering, Vol. 17, Issue 1 https://doi.org/10.1007/s11081-015-9286-x	journal	September 2015
A Derivative-Free Trust-Region Algorithm for the Optimization of Functions Smoothed via Gaussian Convolution Using Adaptive Multiple Importance Sampling Maggiar, Alvaro; Wächter, Andreas; Dolinskaya, Irina S. SIAM Journal on Optimization, Vol. 28, Issue 2 https://doi.org/10.1137/15M1031679	journal	January 2018
Positive bases for linear spaces McKinney, Richard L. Transactions of the American Mathematical Society, Vol. 103, Issue 1 https://doi.org/10.1090/S0002-9947-1962-0147879-X	journal	January 1962
Sobolev seminorm of quadratic functions with applications to derivative-free optimization Zhang, Zaikun Mathematical Programming, Vol. 146, Issue 1-2 https://doi.org/10.1007/s10107-013-0679-3	journal	May 2013
Use of quadratic models with mesh-adaptive direct search for constrained black box optimization Conn, Andrew R.; Le Digabel, Sébastien Optimization Methods and Software, Vol. 28, Issue 1 https://doi.org/10.1080/10556788.2011.623162	journal	February 2013
A Globally Convergent Filter Method for Nonlinear Programming Gonzaga, Clóvis C.; Karas, Elizabeth; Vanti, Márcia SIAM Journal on Optimization, Vol. 14, Issue 3 https://doi.org/10.1137/S1052623401399320	journal	January 2004
Benchmarking Derivative-Free Optimization Algorithms Moré, Jorge J.; Wild, Stefan M. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/080724083	journal	January 2009
Stochastic Estimation of the Maximum of a Regression Function Kiefer, J.; Wolfowitz, J. The Annals of Mathematical Statistics, Vol. 23, Issue 3 https://doi.org/10.1214/aoms/1177729392	journal	September 1952
Wedge trust region methods for derivative free optimization Marazzi, Marcelo; Nocedal, Jorge Mathematical Programming, Vol. 91, Issue 2 https://doi.org/10.1007/s101070100264	journal	January 2002
Convergence of the Restricted Nelder--Mead Algorithm in Two Dimensions Lagarias, Jeffrey C.; Poonen, Bjorn; Wright, Margaret H. SIAM Journal on Optimization, Vol. 22, Issue 2 https://doi.org/10.1137/110830150	journal	January 2012
Using QR Decomposition to Obtain a New Instance of Mesh Adaptive Direct Search with Uniformly Distributed Polling Directions Van Dyke, Benjamin; Asaki, Thomas J. Journal of Optimization Theory and Applications, Vol. 159, Issue 3 https://doi.org/10.1007/s10957-013-0356-y	journal	June 2013
Reducing the Number of Function Evaluations in Mesh Adaptive Direct Search Algorithms Audet, Charles; Ianni, Andrea; Le Digabel, Sébastien SIAM Journal on Optimization, Vol. 24, Issue 2 https://doi.org/10.1137/120895056	journal	January 2014
Objective-derivative-free methods for constrained optimization Lucidi, S.; Sciandrone, M.; Tseng, P. Mathematical Programming, Vol. 92, Issue 1 https://doi.org/10.1007/s101070100266	journal	March 2002
Introduction to Derivative-Free Optimization Conn, Andrew R.; Scheinberg, Katya; Vicente, Luis N. https://doi.org/10.1137/1.9780898718768	book	January 2009
The Levenberg-Marquardt algorithm: Implementation and theory Moré, Jorge J.; Watson, G. A. Numerical Analysis, Vol. 630, p. 105-116 https://doi.org/10.1007/BFb0067700	book	January 1978
Trust Region Methods Conn, Andrew R.; Gould, Nicholas I. M.; Toint, Philippe L. MOS-SIAM Series on Optimization https://doi.org/10.1137/1.9780898719857	book	January 2000
A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation Powell, M. J. D. Advances in Optimization and Numerical Analysis https://doi.org/10.1007/978-94-015-8330-5_4	book	January 1994
A generating set search method using curvature information Frimannslund, Lennart; Steihaug, Trond Computational Optimization and Applications, Vol. 38, Issue 1 https://doi.org/10.1007/s10589-007-9038-8	journal	May 2007
Derivative-Free Optimization Via Proximal Point Methods Hare, W. L.; Lucet, Y. Journal of Optimization Theory and Applications, Vol. 160, Issue 1 https://doi.org/10.1007/s10957-013-0354-0	journal	June 2013
Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization Garmanjani, R.; Vicente, L. N. IMA Journal of Numerical Analysis, Vol. 33, Issue 3 https://doi.org/10.1093/imanum/drs027	journal	October 2012
Developments of NEWUOA for minimization without derivatives Powell, M. J. D. IMA Journal of Numerical Analysis, Vol. 28, Issue 4 https://doi.org/10.1093/imanum/drm047	journal	February 2008
A Progressive Barrier for Derivative-Free Nonlinear Programming Audet, Charles; Dennis, J. E. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/070692662	journal	January 2009
Rates of Convergence for Stochastic Approximation Type Algorithms Kushner, Harold J.; Huang, Hai SIAM Journal on Control and Optimization, Vol. 17, Issue 5 https://doi.org/10.1137/0317043	journal	September 1979
Estimating Tangent and Normal Cones Without Calculus Hare, W. L.; Lewis, A. S. Mathematics of Operations Research, Vol. 30, Issue 4 https://doi.org/10.1287/moor.1050.0163	journal	November 2005
Pattern Search Algorithms for Mixed Variable Programming Audet, Charles; Dennis, J. E. SIAM Journal on Optimization, Vol. 11, Issue 3 https://doi.org/10.1137/S1052623499352024	journal	January 2001
Theory of Positive Linear Dependence Davis, Chandler American Journal of Mathematics, Vol. 76, Issue 4 https://doi.org/10.2307/2372648	journal	October 1954
Exploiting problem structure in pattern search methods for unconstrained optimization Price, C. J.; Toint, Ph. L. Optimization Methods and Software, Vol. 21, Issue 3 https://doi.org/10.1080/10556780500137116	journal	June 2006
A polynomial case of convex integer quadratic programming problems with box integer constraints Liu, Chunli; Gao, Jianjun Journal of Global Optimization, Vol. 62, Issue 4 https://doi.org/10.1007/s10898-014-0263-2	journal	January 2015
Pattern Search Methods for Linearly Constrained Minimization Lewis, Robert Michael; Torczon, Virginia SIAM Journal on Optimization, Vol. 10, Issue 3 https://doi.org/10.1137/S1052623497331373	journal	January 2000
Sequential Penalty Derivative-Free Methods for Nonlinear Constrained Optimization Liuzzi, Giampaolo; Lucidi, Stefano; Sciandrone, Marco SIAM Journal on Optimization, Vol. 20, Issue 5 https://doi.org/10.1137/090750639	journal	January 2010
A derivative-free trust-funnel method for equality-constrained nonlinear optimization Sampaio, Ph. R.; Toint, Ph. L. Computational Optimization and Applications, Vol. 61, Issue 1 https://doi.org/10.1007/s10589-014-9715-3	journal	November 2014
Complexity and global rates of trust-region methods based on probabilistic models Gratton, Serge; Royer, Clément W.; Vicente, Luís N. IMA Journal of Numerical Analysis, Vol. 38, Issue 3 https://doi.org/10.1093/imanum/drx043	journal	August 2017
Nonmonotone derivative-free methods for nonlinear equations Grippo, L.; Sciandrone, M. Computational Optimization and Applications, Vol. 37, Issue 3 https://doi.org/10.1007/s10589-007-9028-x	journal	March 2007
Numerical experience with a derivative-free trust-funnel method for nonlinear optimization problems with general nonlinear constraints Sampaio, Ph. R.; Toint, Ph. L. Optimization Methods and Software, Vol. 31, Issue 3 https://doi.org/10.1080/10556788.2015.1135919	journal	February 2016
Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization Bandeira, A. S.; Scheinberg, K.; Vicente, L. N. Mathematical Programming, Vol. 134, Issue 1 https://doi.org/10.1007/s10107-012-0578-z	journal	July 2012
Derivative-Free Robust Optimization for Circuit Design Ciccazzo, Angelo; Latorre, Vittorio; Liuzzi, Giampaolo Journal of Optimization Theory and Applications, Vol. 164, Issue 3 https://doi.org/10.1007/s10957-013-0441-2	journal	October 2013
Conditional gradient type methods for composite nonlinear and stochastic optimization Ghadimi, Saeed Mathematical Programming, Vol. 173, Issue 1-2 https://doi.org/10.1007/s10107-017-1225-5	journal	January 2018
Simulation optimization: a review of algorithms and applications Amaran, Satyajith; Sahinidis, Nikolaos V.; Sharda, Bikram Annals of Operations Research, Vol. 240, Issue 1 https://doi.org/10.1007/s10479-015-2019-x	journal	September 2015
Asymptotic Distribution of Stochastic Approximation Procedures Sacks, Jerome The Annals of Mathematical Statistics, Vol. 29, Issue 2 https://doi.org/10.1214/aoms/1177706619	journal	June 1958
Calibration by optimization without using derivatives Lazar, Markus; Jarre, Florian Optimization and Engineering, Vol. 17, Issue 4 https://doi.org/10.1007/s11081-016-9324-3	journal	July 2016
Using simplex gradients of nonsmooth functions in direct search methods Custodio, A. L.; Dennis, J. E.; Vicente, L. N. IMA Journal of Numerical Analysis, Vol. 28, Issue 4 https://doi.org/10.1093/imanum/drn045	journal	February 2008
GOSAC: global optimization with surrogate approximation of constraints Müller, Juliane; Woodbury, Joshua D. Journal of Global Optimization, Vol. 69, Issue 1 https://doi.org/10.1007/s10898-017-0496-y	journal	January 2017
Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation Conn, A. R.; Scheinberg, K.; Vicente, L. N. IMA Journal of Numerical Analysis, Vol. 28, Issue 4 https://doi.org/10.1093/imanum/drn046	journal	February 2008
On the Lagrange functions of quadratic models that are defined by interpolation ^* Powell, M. J. D. Optimization Methods and Software, Vol. 16, Issue 1-4 https://doi.org/10.1080/10556780108805839	journal	January 2001
Convergence Properties of the Nelder--Mead Simplex Method in Low Dimensions Lagarias, Jeffrey C.; Reeds, James A.; Wright, Margaret H. SIAM Journal on Optimization, Vol. 9, Issue 1 https://doi.org/10.1137/S1052623496303470	journal	January 1998
Global convergence rate analysis of unconstrained optimization methods based on probabilistic models Cartis, C.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1137-4	journal	April 2017
Function Minimization by Interpolation in a Data Table Winfield, D. IMA Journal of Applied Mathematics, Vol. 12, Issue 3 https://doi.org/10.1093/imamat/12.3.339	journal	January 1973
Budget-Dependent Convergence Rate of Stochastic Approximation L'Ecuyer, Pierre; Yin, George SIAM Journal on Optimization, Vol. 8, Issue 1 https://doi.org/10.1137/S1052623495270723	journal	February 1998
Bilevel direct search method for leader–follower problems and application in health insurance Zhang, Dali; Lin, Gui-Hua Computers & Operations Research, Vol. 41 https://doi.org/10.1016/j.cor.2012.12.005	journal	January 2014
On the Global Convergence of Derivative-Free Methods for Unconstrained Optimization Lucidi, Stefano; Sciandrone, Marco SIAM Journal on Optimization, Vol. 13, Issue 1 https://doi.org/10.1137/S1052623497330392	journal	January 2002
A class of derivative-free nonmonotone optimization algorithms employing coordinate rotations and gradient approximations Grippo, L.; Rinaldi, F. Computational Optimization and Applications, Vol. 60, Issue 1 https://doi.org/10.1007/s10589-014-9665-9	journal	June 2014
On the geometry phase in model-based algorithms for derivative-free optimization Fasano, Giovanni; Morales, José Luis; Nocedal, Jorge Optimization Methods and Software, Vol. 24, Issue 1 https://doi.org/10.1080/10556780802409296	journal	February 2009
A Globally Convergent Augmented Lagrangian Pattern Search Algorithm for Optimization with General Constraints and Simple Bounds Lewis, Robert Michael; Torczon, Virginia SIAM Journal on Optimization, Vol. 12, Issue 4 https://doi.org/10.1137/S1052623498339727	journal	January 2002
On trust region methods for unconstrained minimization without derivatives Powell, M. J. D. Mathematical Programming, Vol. 97, Issue 3 https://doi.org/10.1007/s10107-003-0430-6	journal	August 2003
Least Frobenius norm updating of quadratic models that satisfy interpolation conditions Powell, M. J. D. Mathematical Programming, Vol. 100, Issue 1 https://doi.org/10.1007/s10107-003-0490-7	journal	November 2003
Convergence of the Nelder--Mead Simplex Method to a Nonstationary Point McKinnon, K. I. M. SIAM Journal on Optimization, Vol. 9, Issue 1 https://doi.org/10.1137/S1052623496303482	journal	January 1998
Robust Optimization for Unconstrained Simulation-Based Problems Bertsimas, Dimitris; Nohadani, Omid; Teo, Kwong Meng Operations Research, Vol. 58, Issue 1 https://doi.org/10.1287/opre.1090.0715	journal	February 2010
Pure adaptive search in monte carlo optimization Patel, Nitin R.; Smith, Robert L.; Zabinsky, Zelda B. Mathematical Programming, Vol. 43, Issue 1-3 https://doi.org/10.1007/BF01582296	journal	January 1989
Stochastic Trust-Region Response-Surface Method (STRONG)—A New Response-Surface Framework for Simulation Optimization Chang, Kuo-Hao; Hong, L. Jeff; Wan, Hong INFORMS Journal on Computing, Vol. 25, Issue 2 https://doi.org/10.1287/ijoc.1120.0498	journal	May 2013
Stochastic global optimization methods part I: Clustering methods Rinnooy Kan, A. H. G.; Timmer, G. T. Mathematical Programming, Vol. 39, Issue 1 https://doi.org/10.1007/BF02592070	journal	September 1987
Implementing the Nelder-Mead simplex algorithm with adaptive parameters Gao, Fuchang; Han, Lixing Computational Optimization and Applications, Vol. 51, Issue 1 https://doi.org/10.1007/s10589-010-9329-3	journal	May 2010
Stochastic global optimization methods part II: Multi level methods Rinnooy Kan, A. H. G.; Timmer, G. T. Mathematical Programming, Vol. 39, Issue 1 https://doi.org/10.1007/BF02592071	journal	September 1987
Efficient solution of quadratically constrained quadratic subproblems within the mesh adaptive direct search algorithm Amaioua, Nadir; Audet, Charles; Conn, Andrew R. European Journal of Operational Research, Vol. 268, Issue 1 https://doi.org/10.1016/j.ejor.2017.10.058	journal	July 2018
Nonlinear programming without a penalty function or a filter Gould, N. I. M.; Toint, Ph. L. Mathematical Programming, Vol. 122, Issue 1 https://doi.org/10.1007/s10107-008-0244-7	journal	September 2008
Analysis of Generalized Pattern Searches Audet, Charles; Dennis, J. E. SIAM Journal on Optimization, Vol. 13, Issue 3 https://doi.org/10.1137/S1052623400378742	journal	January 2002
Convergence of Mesh Adaptive Direct Search to Second‐Order Stationary Points Abramson, Mark A.; Audet, Charles SIAM Journal on Optimization, Vol. 17, Issue 2 https://doi.org/10.1137/050638382	journal	January 2006
Direct Search Based on Probabilistic Descent Gratton, S.; Royer, C. W.; Vicente, L. N. SIAM Journal on Optimization, Vol. 25, Issue 3 https://doi.org/10.1137/140961602	journal	January 2015
Detection and Remediation of Stagnation in the Nelder--Mead Algorithm Using a Sufficient Decrease Condition Kelley, C. T. SIAM Journal on Optimization, Vol. 10, Issue 1 https://doi.org/10.1137/S1052623497315203	journal	January 1999
Scientific uses of the MANIAC Anderson, H. L. Journal of Statistical Physics, Vol. 43, Issue 5-6 https://doi.org/10.1007/BF02628301	journal	June 1986
Discrete Gradient Method: Derivative-Free Method for Nonsmooth Optimization Bagirov, A. M.; Karasözen, B.; Sezer, M. Journal of Optimization Theory and Applications, Vol. 137, Issue 2 https://doi.org/10.1007/s10957-007-9335-5	journal	December 2007
A Class of Trust-Region Methods for Parallel Optimization Hough, P. D.; Meza, J. C. SIAM Journal on Optimization, Vol. 13, Issue 1 https://doi.org/10.1137/S1052623498343799	journal	January 2002
SO-I: a surrogate model algorithm for expensive nonlinear integer programming problems including global optimization applications Müller, Juliane; Shoemaker, Christine A.; Piché, Robert Journal of Global Optimization, Vol. 59, Issue 4 https://doi.org/10.1007/s10898-013-0101-y	journal	September 2013
Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales Blanchet, Jose; Cartis, Coralia; Menickelly, Matt INFORMS Journal on Optimization, Vol. 1, Issue 2 https://doi.org/10.1287/ijoo.2019.0016	journal	April 2019
Multivariate Stochastic Process Models for Correlated Responses of Mixed Type Pourmohamad, Tony; Lee, Herbert K. H. Bayesian Analysis, Vol. 11, Issue 3 https://doi.org/10.1214/15-BA976	journal	September 2016
Google Vizier: A Service for Black-Box Optimization Golovin, Daniel; Solnik, Benjamin; Moitra, Subhodeep KDD '17: The 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining https://doi.org/10.1145/3097983.3098043	conference	August 2017
Derivative free methodologies for circuit worst case analysis Latorre, Vittorio; Habal, Husni; Graeb, Helmut Optimization Letters, Vol. 13, Issue 7 https://doi.org/10.1007/s11590-018-1364-5	journal	December 2018
Recent progress in unconstrained nonlinear optimization without derivatives Conn, A. R.; Scheinberg, K.; Toint, Ph. L. Mathematical Programming, Vol. 79, Issue 1-3 https://doi.org/10.1007/BF02614326	journal	October 1997
Surrogate Optimization of Computationally Expensive Black-Box Problems with Hidden Constraints Müller, Juliane; Day, Marcus INFORMS Journal on Computing, Vol. 31, Issue 4 https://doi.org/10.1287/ijoc.2018.0864	journal	October 2019
Stochastic root finding via retrospective approximation Chen, Huifen; Schmeiser, Bruce W. IIE Transactions, Vol. 33, Issue 3 https://doi.org/10.1080/07408170108936827	journal	March 2001

Similar Records

Sign-OPT: A Query-Efficient Hard-label Adversarial Attack

Conference · Sun Apr 26 00:00:00 EDT 2020 · OSTI ID:1545343

Cheng, Minhao; Singh, Simranjit; Chen, Patrick; +3 more

A directional Gaussian smoothing optimization method for computational inverse design in nanophotonics

Journal Article · Fri Jan 01 00:00:00 EST 2021 · Materials & Design · OSTI ID:1545343

Zhang, Jiaxin; Bi, Sirui; Zhang, Guannan

A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Journal Article · Wed Aug 11 00:00:00 EDT 2021 · Journal of Mechanical Design · OSTI ID:1545343

Biswas, Arpan; Fuentes, Claudio; Hoyle, Christopher

Related Subjects

97 MATHEMATICS AND COMPUTING
Optimization

Title: Derivative-free optimization methods

Citation Formats

References (281)

Similar Records

Related Subjects