Stochastic projective splitting

Johnstone, Patrick R.; Eckstein, Jonathan; Flynn, Thomas; Yoo, Shinjae

doi:10.1007/s10589-023-00528-6

Title: Stochastic projective splitting

Journal Article · 23 September 2023 · Computational Optimization and Applications

DOI: https://doi.org/10.1007/s10589-023-00528-6 · OSTI ID:2007522

Johnstone, Patrick R. ^[1]; Eckstein, Jonathan ^[2]; Flynn, Thomas ^[1]; Yoo, Shinjae ^[1]

Brookhaven National Laboratory (BNL), Upton, NY (United States)
Rutgers University, Newark, NJ (United States)

Here, we present a new, stochastic variant of the projective splitting (PS) family of algorithms for inclusion problems involving the sum of any finite number of maximal monotone operators. This new variant uses a stochastic oracle to evaluate one of the operators, which is assumed to be Lipschitz continuous, and (deterministic) resolvents to process the remaining operators. Our proposal is the first version of PS with such stochastic capabilities. We envision the primary application being machine learning (ML) problems, with the method’s stochastic features facilitating “mini-batch” sampling of datasets. Since it uses a monotone operator formulation, the method can handle not only Lipschitz-smooth loss minimization, but also min–max and noncooperative game formulations, with better convergence properties than the gradient descent-ascent methods commonly applied in such settings. The proposed method can handle any number of constraints and nonsmooth regularizers via projection and proximal operators. We prove almost-sure convergence of the iterates to a solution and a convergence rate result for the expected residual, and close with numerical experiments on a distributionally robust sparse logistic regression problem.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0012704

OSTI ID:: 2007522

Report Number(s):: BNL--224851-2023-JAAM

Journal Information:: Computational Optimization and Applications, Journal Name: Computational Optimization and Applications Journal Issue: 2 Vol. 87; ISSN 0926-6003

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (28)

Proximal Splitting Methods in Signal Processing Combettes, Patrick L.; Pesquet, Jean-Christophe Springer Optimization and Its Applications https://doi.org/10.1007/978-1-4419-9569-8_10	book	January 2011
Large-Scale Machine Learning with Stochastic Gradient Descent Bottou, Léon Proceedings of COMPSTAT'2010 https://doi.org/10.1007/978-3-7908-2604-3_16	book	January 2010
Asynchronous block-iterative primal-dual decomposition methods for monotone inclusions Combettes, Patrick L.; Eckstein, Jonathan Mathematical Programming, Vol. 168, Issue 1-2 https://doi.org/10.1007/s10107-016-1044-0	journal	July 2016
Projective splitting with forward steps Johnstone, Patrick R.; Eckstein, Jonathan Mathematical Programming, Vol. 191, Issue 2 https://doi.org/10.1007/s10107-020-01565-3	journal	September 2020
Single-forward-step projective splitting: exploiting cocoercivity Johnstone, Patrick R.; Eckstein, Jonathan Computational Optimization and Applications, Vol. 78, Issue 1 https://doi.org/10.1007/s10589-020-00238-3	journal	November 2020
Forward-reflected-backward method with variance reduction Alacaoglu, Ahmet; Malitsky, Yura; Cevher, Volkan Computational Optimization and Applications, Vol. 80, Issue 2 https://doi.org/10.1007/s10589-021-00305-3	journal	August 2021
A Simplified Form of Block-Iterative Operator Splitting and an Asynchronous Algorithm Resembling the Multi-Block Alternating Direction Method of Multipliers Eckstein, Jonathan Journal of Optimization Theory and Applications, Vol. 173, Issue 1 https://doi.org/10.1007/s10957-017-1074-7	journal	February 2017
Primal-Dual Splitting Algorithm for Solving Inclusions with Mixtures of Composite, Lipschitzian, and Parallel-Sum Type Monotone Operators Combettes, Patrick L.; Pesquet, Jean-Christophe Set-Valued and Variational Analysis, Vol. 20, Issue 2 https://doi.org/10.1007/s11228-011-0191-y	journal	August 2011
A Three-Operator Splitting Scheme and its Optimization Applications Davis, Damek; Yin, Wotao Set-Valued and Variational Analysis, Vol. 25, Issue 4 https://doi.org/10.1007/s11228-017-0421-z	journal	June 2017
Projective splitting with forward steps only requires continuity Johnstone, Patrick R.; Eckstein, Jonathan Optimization Letters, Vol. 14, Issue 1 https://doi.org/10.1007/s11590-019-01509-7	journal	November 2019
Searching for exotic particles in high-energy physics with deep learning Baldi, P.; Sadowski, P.; Whiteson, D. Nature Communications, Vol. 5, Issue 1 https://doi.org/10.1038/ncomms5308	journal	July 2014
General Projective Splitting Methods for Sums of Maximal Monotone Operators Eckstein, Jonathan; Svaiter, B. F. SIAM Journal on Control and Optimization, Vol. 48, Issue 2 https://doi.org/10.1137/070698816	journal	January 2009
Splitting Algorithms for the Sum of Two Nonlinear Operators Lions, P. L.; Mercier, B. SIAM Journal on Numerical Analysis, Vol. 16, Issue 6 https://doi.org/10.1137/0716071	journal	December 1979
On the Complexity of the Hybrid Proximal Extragradient Method for the Iterates and the Ergodic Mean Monteiro, Renato D. C.; Svaiter, B. F. SIAM Journal on Optimization, Vol. 20, Issue 6 https://doi.org/10.1137/090753127	journal	January 2010
A Monotone+Skew Splitting Model for Composite Monotone Inclusions in Duality Briceño-Arias, Luis M.; Combettes, Patrick L. SIAM Journal on Optimization, Vol. 21, Issue 4 https://doi.org/10.1137/10081602X	journal	October 2011
Solving Coupled Composite Monotone Inclusions by Successive Fejér Approximations of their Kuhn--Tucker Set Alotaibi, Abdullah; Combettes, Patrick L.; Shahzad, Naseer SIAM Journal on Optimization, Vol. 24, Issue 4 https://doi.org/10.1137/130950616	journal	January 2014
Stochastic Quasi-Fejér Block-Coordinate Fixed Point Iterations with Random Sweeping Combettes, Patrick L.; Pesquet, Jean-Christophe SIAM Journal on Optimization, Vol. 25, Issue 2 https://doi.org/10.1137/140971233	journal	January 2015
Optimization Methods for Large-Scale Machine Learning Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge SIAM Review, Vol. 60, Issue 2 https://doi.org/10.1137/16M1080173	journal	January 2018
Convergence Rates for Projective Splitting Johnstone, Patrick R.; Eckstein, Jonathan SIAM Journal on Optimization, Vol. 29, Issue 3 https://doi.org/10.1137/18M1203523	journal	January 2019
A Forward-Backward Splitting Method for Monotone Inclusions Without Cocoercivity Malitsky, Yura; Tam, Matthew K. SIAM Journal on Optimization, Vol. 30, Issue 2 https://doi.org/10.1137/18M1207260	journal	January 2020
A Modified Forward-Backward Splitting Method for Maximal Monotone Mappings Tseng, Paul SIAM Journal on Control and Optimization, Vol. 38, Issue 2 https://doi.org/10.1137/S0363012998338806	journal	January 2000
Prox-Method with Rate of Convergence O (1/ t ) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems Nemirovski, Arkadi SIAM Journal on Optimization, Vol. 15, Issue 1 https://doi.org/10.1137/S1052623403425629	journal	January 2004
LIBSVM: A library for support vector machines Chang, Chih-Chung; Lin, Chih-Jen ACM Transactions on Intelligent Systems and Technology, Vol. 2, Issue 3 https://doi.org/10.1145/1961189.1961199	journal	April 2011
Mitigating Unwanted Biases with Adversarial Learning Zhang, Brian Hu; Lemoine, Blake; Mitchell, Margaret Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society https://doi.org/10.1145/3278721.3278779	conference	December 2018
A Stochastic Approximation Method Robbins, Herbert; Monro, Sutton The Annals of Mathematical Statistics, Vol. 22, Issue 3 https://doi.org/10.1214/aoms/1177729586	journal	September 1951
Minibatch Forward-Backward-Forward Methods for Solving Stochastic Variational Inequalities Boţ, Radu Ioan; Mertikopoulos, Panayotis; Staudigl, Mathias Stochastic Systems, Vol. 11, Issue 2 https://doi.org/10.1287/stsy.2019.0064	journal	June 2021
Convex Optimization: Algorithms and Complexity Bubeck, Sébastien Foundations and Trends® in Machine Learning, Vol. 8, Issue 3-4 https://doi.org/10.1561/2200000050	journal	January 2015
Context-Aware Generative Adversarial Privacy Huang, Chong; Kairouz, Peter; Chen, Xiao Entropy, Vol. 19, Issue 12 https://doi.org/10.3390/e19120656	journal	December 2017

Similar Records

A Note on Euler Approximations for Stochastic Differential Equations with Delay

Journal Article · 2013 · Applied Mathematics and Optimization · OSTI ID:22309136

Gyöngy, Istvan; Sabanis, Sotirios

Adversarial classification via distributional robustness with Wasserstein ambiguity

Journal Article · 2022 · Mathematical Programming · OSTI ID:1861139

Ho-Nguyen, Nam; Wright, Stephen J.

A backward SDE method for uncertainty quantification in deep learning

Journal Article · 2021 · Discrete and Continuous Dynamical Systems - Series S · OSTI ID:1865129

Archibald, Richard; Bao, Feng; Cao, Yanzhao; +1 more

Related Subjects

97 MATHEMATICS AND COMPUTING
convex optimization
monotone inclusions
proximal operator splitting
stochastic gradient descent

Title: Stochastic projective splitting

Citation Formats

References (28)

Similar Records

Related Subjects