Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication

Ballard, Grey; Druinsky, Alex; Knight, Nicholas; Schwartz, Oded

doi:10.1145/2755573.2755613

Title: Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication

Abstract

The performance of parallel algorithms for sparse matrix-matrix multiplication is typically determined by the amount of interprocessor communication performed, which in turn depends on the nonzero structure of the input matrices. In this paper, we characterize the communication cost of a sparse matrix-matrix multiplication algorithm in terms of the size of a cut of an associated hypergraph that encodes the computation for a given input nonzero structure. Obtaining an optimal algorithm corresponds to solving a hypergraph partitioning problem. Furthermore, our hypergraph model generalizes several existing models for sparse matrix-vector multiplication, and we can leverage hypergraph partitioners developed for that computation to improve application-specific algorithms for multiplying sparse matrices.

Authors:

Ballard, Grey ^[1]; Druinsky, Alex ^[2]; Knight, Nicholas ^[3]; Schwartz, Oded ^[4]

Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Univ. of California, Berkeley, CA (United States)
Hebrew Univ., Jerusalem (Israel)

Publication Date:: Thu Jan 01 00:00:00 EST 2015

Research Org.:: Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1303161

Report Number(s):: SAND-2016-4718J
640462

Grant/Contract Number:: AC04-94AL85000

Resource Type:: Accepted Manuscript

Journal Name:: ACM Transactions on Parallel Computing

Additional Journal Information:: Journal Name: ACM Transactions on Parallel Computing

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Ballard, Grey, Druinsky, Alex, Knight, Nicholas, and Schwartz, Oded. Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication.  United States: N. p., 2015. 
Web.  doi:10.1145/2755573.2755613.

Copy to clipboard


                    Ballard, Grey, Druinsky, Alex, Knight, Nicholas, & Schwartz, Oded. Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication.  United States.  https://doi.org/10.1145/2755573.2755613

Copy to clipboard


                    Ballard, Grey, Druinsky, Alex, Knight, Nicholas, and Schwartz, Oded. Thu .  
"Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication".  United States.  https://doi.org/10.1145/2755573.2755613.  https://www.osti.gov/servlets/purl/1303161.

Copy to clipboard


                    
@article{osti_1303161,

  title        = {Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication},

  author       = {Ballard, Grey and Druinsky, Alex and Knight, Nicholas and Schwartz, Oded},

  abstractNote = {The performance of parallel algorithms for sparse matrix-matrix multiplication is typically determined by the amount of interprocessor communication performed, which in turn depends on the nonzero structure of the input matrices. In this paper, we characterize the communication cost of a sparse matrix-matrix multiplication algorithm in terms of the size of a cut of an associated hypergraph that encodes the computation for a given input nonzero structure. Obtaining an optimal algorithm corresponds to solving a hypergraph partitioning problem. Furthermore, our hypergraph model generalizes several existing models for sparse matrix-vector multiplication, and we can leverage hypergraph partitioners developed for that computation to improve application-specific algorithms for multiplying sparse matrices.},

  doi          = {10.1145/2755573.2755613},

  journal      = {ACM Transactions on Parallel Computing},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {Thu Jan 01 00:00:00 EST 2015},

  month        = {Thu Jan 01 00:00:00 EST 2015}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1145/2755573.2755613

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 12 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Communication optimal parallel multiplication of sparse random matrices
conference, January 2013

Ballard, Grey; Buluc, Aydin; Demmel, James
Proceedings of the 25th ACM symposium on Parallelism in algorithms and architectures - SPAA '13
DOI: 10.1145/2486159.2486196

Hypergraph Partitioning for Automatic Memory Hierarchy Management
conference, November 2006

Krishnamoorthy, Sriram; Catalyurek, Umit; Nieplocha, Jarek
ACM/IEEE SC 2006 Conference (SC'06)
DOI: 10.1109/SC.2006.36

Simultaneous Input and Output Matrix Partitioning for Outer-Product--Parallel Sparse Matrix-Matrix Multiplication
journal, January 2014

Akbudak, Kadir; Aykanat, Cevdet
SIAM Journal on Scientific Computing, Vol. 36, Issue 5
DOI: 10.1137/13092589X

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid
report, September 2015

Ballard, Grey Malone; Hu, Jonathan Joseph; Siefert, Christopher
DOI: 10.2172/1504845

Graph Algorithms
book, January 1990

Lengauer, Thomas
Combinatorial Algorithms for Integrated Circuit Layout
DOI: 10.1007/978-3-322-92106-2_3

Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments
journal, January 2012

Buluç, Aydin; Gilbert, John R.
SIAM Journal on Scientific Computing, Vol. 34, Issue 4
DOI: 10.1137/110848244

On Two-Dimensional Sparse Matrix Partitioning: Models, Methods, and a Recipe
journal, January 2010

Çatalyürek, Ümt V.; Aykanat, Cevdet; Uçar, Bora
SIAM Journal on Scientific Computing, Vol. 32, Issue 2
DOI: 10.1137/080737770

Data management and query---Hypergraph partitioning for automatic memory hierarchy management
conference, January 2006

Krishnamoorthy, Sriram; Catalyurek, Umit; Nieplocha, Jarek
Proceedings of the 2006 ACM/IEEE conference on Supercomputing - SC '06
DOI: 10.1145/1188455.1188558

Works referencing / citing this record:

Embedded pattern formation by asynchronous robots without chirality
journal, May 2018

Cicerone, Serafino; Di Stefano, Gabriele; Navarra, Alfredo
Distributed Computing, Vol. 32, Issue 4
DOI: 10.1007/s00446-018-0333-7

Distributed-Memory Tensor Completion for Generalized Loss Functions in Python using New Sparse Tensor Kernels
preprint, January 2019

Singh, Navjot; Zhang, Zecheng; Wu, Xiaoxiao
arXiv
DOI: 10.48550/arxiv.1910.02371

Optimizing High Performance Markov Clustering for Pre-Exascale Architectures
text, January 2020

Selvitopi, Oguz; Hussain, Md Taufique; Azad, Ariful
arXiv
DOI: 10.48550/arxiv.2002.10083

Simulating Population Protocols in Sub-Constant Time per Interaction
text, January 2020

Berenbrink, Petra; Hammer, David; Kaaser, Dominik
Schloss Dagstuhl - Leibniz-Zentrum für Informatik
DOI: 10.4230/lipics.esa.2020.16

Optimizing partitioned CSR-based SpGEMM on the Sunway TaihuLight
journal, March 2019

Chen, Yuedan; Xiao, Guoqing; Yang, Wangdong
Neural Computing and Applications, Vol. 32, Issue 10
DOI: 10.1007/s00521-019-04121-z

Privacy-Preserving Data Analytics
book, January 2018

Quoc, Do Le; Beck, Martin; Bhatotia, Pramod
Encyclopedia of Big Data Technologies
DOI: 10.1007/978-3-319-63962-8_152-1

A study on using uncertain time series matching algorithms for MapReduce applications: STATISTICAL PATTERN MATCHING ALGORITHMS FOR MAPREDUCE APPLICATIONS
journal, July 2012

Rizvandi, Nikzad Babaii; Taheri, Javid; Moraveji, Reza
Concurrency and Computation: Practice and Experience, Vol. 25, Issue 12
DOI: 10.1002/cpe.2895

Large-Scale Indexing, Discovery, and Ranking for the Internet of Things (IoT)
journal, June 2018

Fathy, Yasmin; Barnaghi, Payam; Tafazolli, Rahim
ACM Computing Surveys, Vol. 51, Issue 2
DOI: 10.1145/3154525

A novel robust on-line protocol for load-balancing in structured peer-to-peer systems
journal, July 2012

Tsatsanifos, George; Samoladas, Vasilis
Computing, Vol. 94, Issue 8-10
DOI: 10.1007/s00607-012-0199-7

Distributed MST and Routing in Almost Mixing Time
conference, July 2017

Ghaffari, Mohsen; Kuhn, Fabian; Su, Hsin-Hao
PODC '17: ACM Symposium on Principles of Distributed Computing, Proceedings of the ACM Symposium on Principles of Distributed Computing
DOI: 10.1145/3087801.3087827

Hypergraph Partitioning for Sparse Matrix-Matrix Multiplication
preprint, January 2016

Ballard, Grey; Druinsky, Alex; Knight, Nicholas
arXiv
DOI: 10.48550/arxiv.1603.05627

Partitioning Models for Scaling Parallel Sparse Matrix-Matrix Multiplication
journal, April 2018

Akbudak, Kadir; Selvitopi, Oguz; Aykanat, Cevdet
ACM Transactions on Parallel Computing, Vol. 4, Issue 3
DOI: 10.1145/3155292

An Analysis Framework for Distributed Hierarchical Directories
journal, June 2013

Sharma, Gokarna; Busch, Costas
Algorithmica, Vol. 71, Issue 2
DOI: 10.1007/s00453-013-9803-2

On Counting the Population Size
conference, July 2019

Berenbrink, Petra; Kaaser, Dominik; Radzik, Tomasz
PODC '19: ACM Symposium on Principles of Distributed Computing, Proceedings of the 2019 ACM Symposium on Principles of Distributed Computing
DOI: 10.1145/3293611.3331631

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid
report, September 2015

Ballard, Grey Malone; Hu, Jonathan Joseph; Siefert, Christopher
DOI: 10.2172/1504845

Numerical algorithms for high-performance computational science
journal, January 2020

Dongarra, Jack; Grigori, Laura; Higham, Nicholas J.
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 378, Issue 2166
DOI: 10.1098/rsta.2019.0066

Similar Records in DOE PAGES and OSTI.GOV collections:

A nested dissection approach to sparse matrix partitioning for parallel computations.

Journal Article Wolf, Michael M ; Boman, Erik Gunnar - Proposed for publication in SIAM Journal on Scientific Computing.

We consider how to distribute sparse matrices among processes to reduce communication costs in parallel sparse matrix computations, specifically, sparse matrix-vector multiplication. Our main contributions are: (i) an exact graph model for communication with general (two-dimensional) matrix distribution, and (ii) a recursive partitioning algorithm based on nested dissection (substructuring). We show that the communication volume is closely linked to vertex separators. We have implemented our algorithm using hypergraph partitioning software to enable a fair comparison with existing methods. We present numerical results for sparse matrices from several application areas, with up to 9 million nonzeros. The results show that ourmore »« less
Partitioning Models for Scaling Parallel Sparse Matrix-Matrix Multiplication

Journal Article Akbudak, Kadir ; Selvitopi, Oguz ; Aykanat, Cevdet - ACM Transactions on Parallel Computing

We investigate outer-product--parallel, inner-product--parallel, and row-by-row-product--parallel formulations of sparse matrix-matrix multiplication (SpGEMM) on distributed memory architectures. For each of these three formulations, we propose a hypergraph model and a bipartite graph model for distributing SpGEMM computations based on one-dimensional (1D) partitioning of input matrices. Here, we also propose a communication hypergraph model for each formulation for distributing communication operations. The computational graph and hypergraph models adopted in the first phase aim at minimizing the total message volume and balancing the computational loads of processors, whereas the communication hypergraph models adopted in the second phase aim at minimizing the total messagemore »« less
https://doi.org/10.1145/3155292

Full Text Available
Communication-Avoiding and Memory-Constrained Sparse Matrix-Matrix Multiplication at Extreme Scale

Journal Article Hussain, Md Taufique ; Selvitopi, Oguz ; Buluc, Aydin ; ... - Proceedings - IEEE International Parallel and Distributed Processing Symposium (IPDPS)

Sparse matrix-matrix multiplication (SpGEMM) is a widely used kernel in various graph, scientific computing and machine learning algorithms. In this paper, we consider SpGEMMs performed on hundreds of thousands of processors generating trillions of nonzeros in the output matrix. Distributed SpGEMM at this extreme scale faces two key challenges: (1) high communication cost and (2) inadequate memory to generate the output. Furthermore, we address these challenges with an integrated communication-avoiding and memory-constrained SpGEMM algorithm that scales to 262,144 cores (more than 1 million hardware threads) and can multiply sparse matrices of any size as long as inputs and a fractionmore »« less
https://doi.org/10.1109/ipdps49936.2021.00018

Full Text Available
Parallel hypergraph partitioning for scientific computing.

Conference Heaphy, Robert ; Devine, Karen Dragon ; Catalyurek, Umit ; ...

Graph partitioning is often used for load balancing in parallel computing, but it is known that hypergraph partitioning has several advantages. First, hypergraphs more accurately model communication volume, and second, they are more expressive and can better represent nonsymmetric problems. Hypergraph partitioning is particularly suited to parallel sparse matrix-vector multiplication, a common kernel in scientific computing. We present a parallel software package for hypergraph (and sparse matrix) partitioning developed at Sandia National Labs. The algorithm is a variation on multilevel partitioning. Our parallel implementation is novel in that it uses a two-dimensional data distribution among processors. We present empirical resultsmore »« less
MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization

Journal Article Kannan, Ramakrishnan ; Ballard, Grey ; Park, Haesun - IEEE Transactions on Knowledge and Data Engineering

Non-negative matrix factorization (NMF) is the problem of determining two non-negative low rank factors W and H, for the given input matrix A, such that A≈WH. NMF is a useful tool for many applications in different domains such as topic modeling in text mining, background separation in video analysis, and community detection in social networks. Despite its popularity in the data mining community, there is a lack of efficient parallel algorithms to solve the problem for big data sets. The main contribution of this work is a new, high-performance parallel computational framework for a broad class of NMF algorithms thatmore »« less
Cited by 17
https://doi.org/10.1109/TKDE.2017.2767592

Full Text Available

Similar Records

Title: Brief announcement: Hypergraph parititioning for parallel sparse matrix-matrix multiplication

Abstract

Citation Formats

Communication optimal parallel multiplication of sparse random matrices conference, January 2013

Hypergraph Partitioning for Automatic Memory Hierarchy Management conference, November 2006

Simultaneous Input and Output Matrix Partitioning for Outer-Product--Parallel Sparse Matrix-Matrix Multiplication journal, January 2014

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid report, September 2015

Graph Algorithms book, January 1990

Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments journal, January 2012

On Two-Dimensional Sparse Matrix Partitioning: Models, Methods, and a Recipe journal, January 2010

Data management and query---Hypergraph partitioning for automatic memory hierarchy management conference, January 2006

Embedded pattern formation by asynchronous robots without chirality journal, May 2018

Distributed-Memory Tensor Completion for Generalized Loss Functions in Python using New Sparse Tensor Kernels preprint, January 2019

Optimizing High Performance Markov Clustering for Pre-Exascale Architectures text, January 2020

Simulating Population Protocols in Sub-Constant Time per Interaction text, January 2020

Optimizing partitioned CSR-based SpGEMM on the Sunway TaihuLight journal, March 2019

Privacy-Preserving Data Analytics book, January 2018

A study on using uncertain time series matching algorithms for MapReduce applications: STATISTICAL PATTERN MATCHING ALGORITHMS FOR MAPREDUCE APPLICATIONS journal, July 2012

Large-Scale Indexing, Discovery, and Ranking for the Internet of Things (IoT) journal, June 2018

A novel robust on-line protocol for load-balancing in structured peer-to-peer systems journal, July 2012

Distributed MST and Routing in Almost Mixing Time conference, July 2017

Hypergraph Partitioning for Sparse Matrix-Matrix Multiplication preprint, January 2016

Partitioning Models for Scaling Parallel Sparse Matrix-Matrix Multiplication journal, April 2018

An Analysis Framework for Distributed Hierarchical Directories journal, June 2013

On Counting the Population Size conference, July 2019

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid report, September 2015

Numerical algorithms for high-performance computational science journal, January 2020

Communication optimal parallel multiplication of sparse random matrices
conference, January 2013

Hypergraph Partitioning for Automatic Memory Hierarchy Management
conference, November 2006

Simultaneous Input and Output Matrix Partitioning for Outer-Product--Parallel Sparse Matrix-Matrix Multiplication
journal, January 2014

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid
report, September 2015

Graph Algorithms
book, January 1990

Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments
journal, January 2012

On Two-Dimensional Sparse Matrix Partitioning: Models, Methods, and a Recipe
journal, January 2010

Data management and query---Hypergraph partitioning for automatic memory hierarchy management
conference, January 2006

Embedded pattern formation by asynchronous robots without chirality
journal, May 2018

Distributed-Memory Tensor Completion for Generalized Loss Functions in Python using New Sparse Tensor Kernels
preprint, January 2019

Optimizing High Performance Markov Clustering for Pre-Exascale Architectures
text, January 2020

Simulating Population Protocols in Sub-Constant Time per Interaction
text, January 2020

Optimizing partitioned CSR-based SpGEMM on the Sunway TaihuLight
journal, March 2019

Privacy-Preserving Data Analytics
book, January 2018

A study on using uncertain time series matching algorithms for MapReduce applications: STATISTICAL PATTERN MATCHING ALGORITHMS FOR MAPREDUCE APPLICATIONS
journal, July 2012

Large-Scale Indexing, Discovery, and Ranking for the Internet of Things (IoT)
journal, June 2018

A novel robust on-line protocol for load-balancing in structured peer-to-peer systems
journal, July 2012

Distributed MST and Routing in Almost Mixing Time
conference, July 2017

Hypergraph Partitioning for Sparse Matrix-Matrix Multiplication
preprint, January 2016

Partitioning Models for Scaling Parallel Sparse Matrix-Matrix Multiplication
journal, April 2018

An Analysis Framework for Distributed Hierarchical Directories
journal, June 2013

On Counting the Population Size
conference, July 2019

Reducing Communication Costs for Sparse Matrix Multiplication within Algebraic Multigrid
report, September 2015

Numerical algorithms for high-performance computational science
journal, January 2020