Matrix Factorizations at Scale: a Comparison of Scientific Data Analytics in Spark and C+MPI Using Three Case Studies

Conference · Thu May 12 00:00:00 EDT 2016

OSTI ID:1332132

Gittens, Alex; Devarakonda, Aditya; Racah, Evan; Ringenburg, Michael; Gerhardt, Lisa; Kottalam, Jey; Liu, Jialin; Maschhoff, Kristyn; Canon, Shane; Chhugani, Jatin; Sharma, Pramod; Yang, Jiyan; Demmel, James; Harrell, Jim; Krishnamurthy, Venkat; Mahoney, Michael; Prabhat, Mr

We explore the trade-offs of performing linear algebra using Apache Spark, compared to traditional C and MPI implementations on HPC platforms. Spark is designed for data analytics on cluster computing platforms with access to local disks and is optimized for data-parallel tasks. We examine three widely-used and important matrix factorizations: NMF (for physical plausibility), PCA (for its ubiquity) and CX (for data interpretability). We apply these methods to 1.6TB particle physics, 2.2TB and 16TB climate modeling and 1.1TB bioimaging data. The data matrices are tall-and-skinny which enable the algorithms to map conveniently into Spark’s data parallel model. We perform scaling experiments on up to 1600 Cray XC40 nodes, describe the sources of slowdowns, and provide tuning guidance to obtain high performance.

• Research Organization: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: National Energy Research Scientific Computing Division
• OSTI ID: 1332132
• Report Number(s): LBNL-1006428; ir:1006428
• Resource Relation: Conference: 2016 IEEE International Conference on Big Data, Washington DC, USA, 05/12/2016
• Country of Publication: United States
• Language: English