HPC formulations of optimization algorithms for tensor completion

Journal Article · Fri Nov 03 00:00:00 EDT 2017 · Parallel Computing

DOI:https://doi.org/10.1016/j.parco.2017.11.002· OSTI ID:1478749

Smith, Shaden ^[1]; Park, Jongsoo ^[1]; Karypis, George ^[1]

---

1. Univ. of Minnesota, Minneapolis, MN (United States)

Tensor completion is a powerful tool used to estimate or recover missing values in multi-way data. It has seen great success in domains such as product recommendation and healthcare. Tensor completion is most often accomplished via low-rank sparse tensor factorization, a computationally expensive non-convex optimization problem which has only recently been studied in the context of parallel computing. In this work, we study three optimization algorithms that have been successfully applied to tensor completion: alternating least squares (ALS), stochastic gradient descent (SGD), and coordinate descent (CCD++). We explore opportunities for parallelism on shared- and distributed-memory systems and address challenges such as memory- and operation-efficiency, load balance, cache locality, and communication. Among our advancements are a communication-efficient CCD++ algorithm, an ALS algorithm rich in level-3 BLAS routines, and an SGD algorithm which combines stratification with asynchronous communication. Furthermore, we show that introducing randomization during ALS and CCD++ can accelerate convergence. We evaluate our parallel formulations on a variety of real datasets on a modern supercomputer and demonstrate speedups through 16384 cores. Further, these improvements reduce time-to-solution from hours to seconds on real-world datasets. We show that after our optimizations, ALS is advantageous on parallel systems of small-to-moderate scale, while both ALS and CCD++ provide the lowest time-to-solution on large-scale distributed systems.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
• Sponsoring Organization: USDOE Office of Science (SC); National Science Foundation (NSF); US Army Research Office (ARO)
• Grant/Contract Number: AC02-05CH11231
• OSTI ID: 1478749
• Alternate ID(s): OSTI ID: 1549506
• Journal Information: Parallel Computing, Journal Name: Parallel Computing Journal Issue: C Vol. 74; ISSN 0167-8191
• Publisher: Elsevier
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Factorization
Machine learning
Recommender system
Sparse tensor
Tensor completion
Unstructured algorithm