Derivative-free stochastic optimization via adaptive sampling strategies

Bollapragada, Raghu; Karamanli, Cem; Wild, Stefan M.

doi:10.1080/10556788.2025.2547397

Derivative-free stochastic optimization via adaptive sampling strategies

Journal Article · Wed Sep 17 00:00:00 EDT 2025 · Optimization Methods and Software

DOI:https://doi.org/10.1080/10556788.2025.2547397· OSTI ID:2998214

^[1]; ^[1]; ^[2]

Univ. of Texas, Austin, TX (United States)
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Northwestern Univ., Evanston, IL (United States)

Here, in this paper, we present a novel derivative-free framework for solving unconstrained stochastic optimization problems. Many problems in fields ranging from simulation optimization to reinforcement learning to quantum computing involve settings where only stochastic function values are obtained via a zeroth-order oracle, which has no available gradient information and necessitates the usage of derivative-free optimization methodologies. Our approach includes estimating gradients using stochastic function evaluations and integrating adaptive sampling techniques to control the accuracy in these stochastic approximations. Our framework encapsulates several gradient estimation techniques, including standard finite-difference, Gaussian smoothing, sphere smoothing, randomized coordinate finite-difference, and randomized subspace finite-difference methods. We provide theoretical convergence guarantees for our framework and analyze the worst-case iteration and sample complexities associated with each gradient estimation method. Finally, we demonstrate the empirical performance of the methods on logistic regression and nonlinear least squares problems.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF)

Grant/Contract Number:: AC02-05CH11231; SC0019902

OSTI ID:: 2998214

Journal Information:: Optimization Methods and Software, Journal Name: Optimization Methods and Software; ISSN 1055-6788; ISSN 1029-4937

Publisher:: Informa UK LimitedCopyright Statement

Country of Publication:: United States

Language:: English

References (55)

A Guide to Sample Average Approximation Kim, Sujin; Pasupathy, Raghu; Henderson, Shane G. International Series in Operations Research & Management Science https://doi.org/10.1007/978-1-4939-1384-8_8	book	September 2014
Derivative-Free and Blackbox Optimization Audet, Charles; Hare, Warren Springer Series in Operations Research and Financial Engineering https://doi.org/10.1007/978-3-319-68913-5	book	January 2017
Sample size selection in optimization methods for machine learning Byrd, Richard H.; Chin, Gillian M.; Nocedal, Jorge Mathematical Programming, Vol. 134, Issue 1 https://doi.org/10.1007/s10107-012-0572-5	journal	June 2012
Coordinate descent algorithms Wright, Stephen J. Mathematical Programming, Vol. 151, Issue 1 https://doi.org/10.1007/s10107-015-0892-3	journal	March 2015
Global convergence rate analysis of unconstrained optimization methods based on probabilistic models Cartis, C.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1137-4	journal	April 2017
Stochastic optimization using a trust-region method and random models Chen, R.; Menickelly, M.; Scheinberg, K. Mathematical Programming, Vol. 169, Issue 2 https://doi.org/10.1007/s10107-017-1141-8	journal	April 2017
Wedge trust region methods for derivative free optimization Marazzi, Marcelo; Nocedal, Jorge Mathematical Programming, Vol. 91, Issue 2 https://doi.org/10.1007/s101070100264	journal	January 2002
Random Gradient-Free Minimization of Convex Functions Nesterov, Yurii; Spokoiny, Vladimir Foundations of Computational Mathematics, Vol. 17, Issue 2 https://doi.org/10.1007/s10208-015-9296-2	journal	November 2015
A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimization Berahas, Albert S.; Cao, Liyuan; Choromanski, Krzysztof Foundations of Computational Mathematics, Vol. 22, Issue 2 https://doi.org/10.1007/s10208-021-09513-z	journal	May 2021
An adaptive Monte Carlo algorithm for computing mixed logit estimators Bastin, Fabian; Cirillo, Cinzia; Toint, Philippe L. Computational Management Science, Vol. 3, Issue 1 https://doi.org/10.1007/s10287-005-0044-y	journal	January 2006
A stochastic subspace approach to gradient-free optimization in high dimensions Kozak, David; Becker, Stephen; Doostan, Alireza Computational Optimization and Applications, Vol. 79, Issue 2 https://doi.org/10.1007/s10589-021-00271-w	journal	April 2021
Optimization with hidden constraints and embedded Monte Carlo computations Chen, Xiaojun; Kelley, C. T. Optimization and Engineering, Vol. 17, Issue 1 https://doi.org/10.1007/s11081-015-9302-1	journal	December 2015
A derivative-free Gauss–Newton method Cartis, Coralia; Roberts, Lindon Mathematical Programming Computation, Vol. 11, Issue 4 https://doi.org/10.1007/s12532-019-00161-7	journal	May 2019
Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization Bollapragada, Raghu; Wild, Stefan M. Mathematical Programming Computation, Vol. 15, Issue 2 https://doi.org/10.1007/s12532-023-00233-9	journal	March 2023
An adaptive sampling augmented Lagrangian method for stochastic optimization with deterministic constraints Bollapragada, Raghu; Karamanli, Cem; Keith, Brendan Computers & Mathematics with Applications, Vol. 149 https://doi.org/10.1016/j.camwa.2023.09.014	journal	November 2023
Exploration–exploitation tradeoff using variance estimates in multi-armed bandits Audibert, Jean-Yves; Munos, Rémi; Szepesvári, Csaba Theoretical Computer Science, Vol. 410, Issue 19 https://doi.org/10.1016/j.tcs.2009.01.016	journal	April 2009
Derivative-free optimization methods Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. Acta Numerica, Vol. 28 https://doi.org/10.1017/S0962492919000060	journal	May 2019
Sequential Application of Simplex Designs in Optimisation and Evolutionary Operation Spendley, W.; Hext, G. R.; Himsworth, F. R. Technometrics, Vol. 4, Issue 4 https://doi.org/10.1080/00401706.1962.10490033	journal	November 1962
A Simplex Method for Function Minimization Nelder, J. A.; Mead, R. The Computer Journal, Vol. 7, Issue 4 https://doi.org/10.1093/comjnl/7.4.308	journal	January 1965
Adaptive sampling strategies for risk-averse stochastic optimization with constraints Beiser, Florian; Keith, Brendan; Urbainczyk, Simon IMA Journal of Numerical Analysis, Vol. 43, Issue 6 https://doi.org/10.1093/imanum/drac083	journal	January 2023
Constrained and composite optimization via adaptive sampling methods Xie, Yuchen; Bollapragada, Raghu; Byrd, Richard IMA Journal of Numerical Analysis, Vol. 44, Issue 2 https://doi.org/10.1093/imanum/drad020	journal	May 2023
Multivariate stochastic approximation using a simultaneous perturbation gradient approximation Spall, J. C. IEEE Transactions on Automatic Control, Vol. 37, Issue 3 https://doi.org/10.1109/9.119632	journal	March 1992
Optimizing Simulations with Noise-Tolerant Structured Exploration Choromanski, Krzysztof; Iscen, Atil; Sindhwani, Vikas 2018 IEEE International Conference on Robotics and Automation (ICRA) https://doi.org/10.1109/ICRA.2018.8460492	conference	May 2018
Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations Duchi, John C.; Jordan, Michael I.; Wainwright, Martin J. IEEE Transactions on Information Theory, Vol. 61, Issue 5 https://doi.org/10.1109/TIT.2015.2409256	journal	May 2015
Adaptation of the Uobyqa Algorithm for Noisy Functions Deng, Geng; Ferris, Michael Proceedings of the 2006 Winter Simulation Conference https://doi.org/10.1109/WSC.2006.323088	conference	December 2006
On adaptive sampling rules for stochastic recursions Hashemi, Fatemeh S.; Ghosh, Soumyadip; Pasupathy, Raghu Proceedings of the Winter Simulation Conference 2014 https://doi.org/10.1109/WSC.2014.7020221	conference	December 2014
Central Finite-Difference Based Gradient Estimation Methods for Stochastic Optimization Bollapragada, Raghu; Karamanli, Cem; Wild, Stefan M. 2024 Winter Simulation Conference (WSC) https://doi.org/10.1109/WSC63780.2024.10838626	conference	December 2024
Convergence rates of finite difference stochastic approximation algorithms part II: implementation via common random numbers Dai, Liyi SPIE Proceedings https://doi.org/10.1117/12.2228251	conference	May 2016
Stochastic online optimization. Single-point and multi-point non-linear multi-armed bandits. Convex and strongly-convex case Gasnikov, A. V.; Krymova, E. A.; Lagunovskaya, A. A. Automation and Remote Control, Vol. 78, Issue 2 https://doi.org/10.1134/S0005117917020035	journal	February 2017
On the Global Convergence of Trust Region Algorithms Using Inexact Gradient Information Carter, Richard G. SIAM Journal on Numerical Analysis, Vol. 28, Issue 1 https://doi.org/10.1137/0728014	journal	February 1991
Benchmarking Derivative-Free Optimization Algorithms Moré, Jorge J.; Wild, Stefan M. SIAM Journal on Optimization, Vol. 20, Issue 1 https://doi.org/10.1137/080724083	journal	January 2009
Hybrid Deterministic-Stochastic Methods for Data Fitting Friedlander, Michael P.; Schmidt, Mark SIAM Journal on Scientific Computing, Vol. 34, Issue 3 https://doi.org/10.1137/110830629	journal	January 2012
Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming Ghadimi, Saeed; Lan, Guanghui SIAM Journal on Optimization, Vol. 23, Issue 4 https://doi.org/10.1137/120880811	journal	January 2013
On Sampling Rates in Simulation-Based Recursions Pasupathy, Raghu; Glynn, Peter; Ghosh, Soumyadip SIAM Journal on Optimization, Vol. 28, Issue 1 https://doi.org/10.1137/140951679	journal	January 2018
Levenberg--Marquardt Methods Based on Probabilistic Gradient Models and Inexact Subproblem Solution, with Application to Data Assimilation Bergou, E.; Gratton, S.; Vicente, L. N. SIAM/ASA Journal on Uncertainty Quantification, Vol. 4, Issue 1 https://doi.org/10.1137/140974687	journal	January 2016
ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimization Shashaani, Sara; Hashemi, Fatemeh S.; Pasupathy, Raghu SIAM Journal on Optimization, Vol. 28, Issue 4 https://doi.org/10.1137/15M1042425	journal	January 2018
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
Adaptive Sampling Strategies for Stochastic Optimization Bollapragada, Raghu; Byrd, Richard; Nocedal, Jorge SIAM Journal on Optimization, Vol. 28, Issue 4 https://doi.org/10.1137/17M1154679	journal	January 2018
Derivative-Free Optimization of Noisy Functions via Quasi-Newton Methods Berahas, Albert S.; Byrd, Richard H.; Nocedal, Jorge SIAM Journal on Optimization, Vol. 29, Issue 2 https://doi.org/10.1137/18M1177718	journal	January 2019
A Stochastic Line Search Method with Expected Complexity Analysis Paquette, Courtney; Scheinberg, Katya SIAM Journal on Optimization, Vol. 30, Issue 1 https://doi.org/10.1137/18M1216250	journal	January 2020
Global Convergence Rate Analysis of a Generic Line Search Algorithm with Noise Berahas, A. S.; Cao, L.; Scheinberg, K. SIAM Journal on Optimization, Vol. 31, Issue 2 https://doi.org/10.1137/19M1291832	journal	January 2021
Stochastic Trust-Region Algorithm in Random Subspaces with Convergence and Expected Complexity Analyses Dzahini, K. J.; Wild, S. M. SIAM Journal on Optimization, Vol. 34, Issue 3 https://doi.org/10.1137/22M1524072	journal	July 2024
Budget-Dependent Convergence Rate of Stochastic Approximation L'Ecuyer, Pierre; Yin, George SIAM Journal on Optimization, Vol. 8, Issue 1 https://doi.org/10.1137/S1052623495270723	journal	February 1998
LIBSVM: A library for support vector machines Chang, Chih-Chung; Lin, Chih-Jen ACM Transactions on Intelligent Systems and Technology, Vol. 2, Issue 3 https://doi.org/10.1145/1961189.1961199	journal	April 2011
Variable-sample methods for stochastic optimization Homem-De-Mello, Tito ACM Transactions on Modeling and Computer Simulation, Vol. 13, Issue 2 https://doi.org/10.1145/858481.858483	journal	April 2003
CUTEr and SifDec: A constrained and unconstrained testing environment, revisited Gould, Nicholas I. M.; Orban, Dominique; Toint, Philippe L. ACM Transactions on Mathematical Software, Vol. 29, Issue 4 https://doi.org/10.1145/962437.962439	journal	December 2003
Multidimensional Stochastic Approximation Methods Blum, Julius R. The Annals of Mathematical Statistics, Vol. 25, Issue 4 https://doi.org/10.1214/aoms/1177728659	journal	December 1954
Stochastic Estimation of the Maximum of a Regression Function Kiefer, J.; Wolfowitz, J. The Annals of Mathematical Statistics, Vol. 23, Issue 3 https://doi.org/10.1214/aoms/1177729392	journal	September 1952
Stochastic Trust-Region Response-Surface Method (STRONG)—A New Response-Surface Framework for Simulation Optimization Chang, Kuo-Hao; Hong, L. Jeff; Wan, Hong INFORMS Journal on Computing, Vol. 25, Issue 2 https://doi.org/10.1287/ijoc.1120.0498	journal	May 2013
Finite Difference Gradient Approximation: To Randomize or Not? Scheinberg, Katya INFORMS Journal on Computing, Vol. 34, Issue 5 https://doi.org/10.1287/ijoc.2022.1218	journal	September 2022
A Novel Noise-Aware Classical Optimizer for Variational Quantum Algorithms Larson, Jeffrey; Menickelly, Matt; Shi, Jiahao INFORMS Journal on Computing, Vol. 37, Issue 1 https://doi.org/10.1287/ijoc.2024.0578	journal	January 2025
Convergence Rate Analysis of a Stochastic Trust-Region Method via Supermartingales Blanchet, Jose; Cartis, Coralia; Menickelly, Matt INFORMS Journal on Optimization, Vol. 1, Issue 2 https://doi.org/10.1287/ijoo.2019.0016	journal	April 2019
Simulation-Based Optimization with Stochastic Approximation Using Common Random Numbers Kleinman, Nathan L.; Spall, James C.; Naiman, Daniel Q. Management Science, Vol. 45, Issue 11 https://doi.org/10.1287/mnsc.45.11.1570	journal	November 1999
A Retrospective Approximation Approach for Smooth Stochastic Optimization Newton, David; Bollapragada, Raghu; Pasupathy, Raghu Mathematics of Operations Research, Vol. 50, Issue 3 https://doi.org/10.1287/moor.2022.0136	journal	August 2025
Latency considerations for stochastic optimizers in variational quantum algorithms Menickelly, Matt; Ha, Yunsoo; Otten, Matthew Quantum, Vol. 7 https://doi.org/10.22331/q-2023-03-16-949	journal	March 2023

Similar Records

Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization

Journal Article · Mon Mar 13 20:00:00 EDT 2023 · Mathematical Programming Computation · OSTI ID:2001234

Derivative-free optimization methods

Journal Article · Tue Apr 30 20:00:00 EDT 2019 · Acta Numerica · OSTI ID:1545343

Stochastic Trust-Region Algorithm in Random Subspaces with Convergence and Expected Complexity Analyses

Journal Article · Wed Jul 24 20:00:00 EDT 2024 · SIAM Journal on Optimization · OSTI ID:2480304

Related Subjects

Adaptive sampling
common random numbers
derivative-free optimization
gradient approximations

Derivative-free stochastic optimization via adaptive sampling strategies

Citation Formats

References (55)

Similar Records

Related Subjects