skip to main content


Title: Performance analysis of distributed symmetric sparse matrix vector multiplication algorithm for multi-core architectures

In this article, sparse matrix vector multiply (SpMVM) is an important kernel that frequently arises in high performance computing applications. Due to its low arithmetic intensity, several approaches have been proposed in literature to improve its scalability and efficiency in large scale computations. In this paper, our target systems are high end multi-core architectures and we use messaging passing interface + open multiprocessing hybrid programming model for parallelism. We analyze the performance of recently proposed implementation of the distributed symmetric SpMVM, originally developed for large sparse symmetric matrices arising in ab initio nuclear structure calculations. We also study important features of this implementation and compare with previously reported implementations that do not exploit underlying symmetry. Our SpMVM implementations leverage the hybrid paradigm to efficiently overlap expensive communications with computations. Our main comparison criterion is the "CPU core hours" metric, which is the main measure of resource usage on supercomputers. We analyze the effects of topology-aware mapping heuristic using simplified network load model. Furthermore, we have tested the different SpMVM implementations on two large clusters with 3D Torus and Dragonfly topology. Our results show that the distributed SpMVM implementation that exploits matrix symmetry and hides communication yields the best value formore » the "CPU core hours" metric and significantly reduces data movement overheads.« less
 [1] ;  [2] ;  [3] ;  [1] ;  [1]
  1. Iowa State Univ., Ames, IA (United States)
  2. Michigan State Univ., East Lansing, MI (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  3. Iowa State Univ., Ames, IA (United States); Old Dominion Univ., Norfolk, VA (United States)
Publication Date:
Report Number(s):
Journal ID: ISSN 1532-0626
Grant/Contract Number:
0941434; 0904782; 1047772; AC02-07CH11358; SC0008485; FG02-87ER40371; AC02-05CH11231
Accepted Manuscript
Journal Name:
Concurrency and Computation. Practice and Experience
Additional Journal Information:
Journal Volume: 27; Journal Issue: 17; Journal ID: ISSN 1532-0626
Research Org:
Ames Laboratory (AMES), Ames, IA (United States)
Sponsoring Org:
Country of Publication:
United States
97 MATHEMATICS AND COMPUTING; distributed symmetric SpMVM; hybrid MPI/OpenMP parallelism; topology-aware mapping; reduced data movement
OSTI Identifier: