Avoiding Communication in Primal and Dual Block Coordinate Descent Methods

Journal Article · 16 January 2019 · SIAM Journal on Scientific Computing

DOI: https://doi.org/10.1137/17M1134433 · OSTI ID:1544140

Devarakonda, Aditya  ^[1]; Fountoulakis, Kimon ^[1]; Demmel, James ^[1]; Mahoney, Michael W. ^[1]

---

1. Univ. of California, Berkeley, CA (United States)

Primal and dual block coordinate descent methods are iterative methods for solving regularized and unregularized optimization problems. Distributed-memory parallel implementations of these methods have become popular in analyzing large machine learning datasets. However, existing implementations communicate at every iteration, which, on modern data center and supercomputing architectures, often dominates the cost of floating-point computation. Recent results on communication-avoiding Krylov subspace methods suggest that large speedups are possible by re-organizing iterative algorithms to avoid communication. We show how applying similar algorithmic transformations can lead to primal and dual block coordinate descent methods that only communicate every $$s$$ iterations---where $$s$$ is a tuning parameter---instead of every iteration for the regularized least-squares problem. We show that the communication-avoiding variants reduce the number of synchronizations by a factor of $$s$$ on distributed-memory parallel machines without altering the convergence rate and attain strong scaling speedups of up to $$6.1\times$$ over the “standard algorithm" on a Cray XC30 supercomputer.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
• Sponsoring Organization: USDOE
• OSTI ID: 1544140
• Journal Information: SIAM Journal on Scientific Computing, Journal Name: SIAM Journal on Scientific Computing Journal Issue: 1 Vol. 41; ISSN 1064-8275
• Publisher: SIAMCopyright Statement
• Country of Publication: United States
• Language: English