Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware Parallelism

Collom, Gerald; Li, Rui Peng; Bienz, Amanda

doi:10.1145/3624062.3624111

Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware Parallelism

Conference · 12 November 2023 · Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis

DOI:https://doi.org/10.1145/3624062.3624111· OSTI ID:2205620

^[1]; ^[2]; ^[1]

University of New Mexico, Albuquerque, NM (United States)
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Irregular communication often limits both the performance and scalability of parallel applications. Typically, applications individually implement irregular communication as point-to-point, and any optimizations are integrated directly into the application. As a result, these optimizations lack portability. It is difficult to optimize point-to-point messages within MPI, as the interface for single messages provides no information on the collection of all communication to be performed. However, the persistent neighbor collective API, released in the MPI 4 standard, provides an interface for portable optimizations of irregular communication within MPI libraries. This paper presents methods for implementing existing optimizations for irregular communication within neighborhood collectives, analyzes the impact of replacing point-to-point communication in existing codebases such as Hypre BoomerAMG with neighborhood collectives, and finally shows up to a 1.38x speedup on sparse matrix-vector multiplication communication within a BoomerAMG solve through the use of our optimized neighbor collectives. Here, the authors analyze three implementations of persistent neighborhood collectives for Alltoallv: an unoptimized wrapper of standard point-to-point communication, and two locality-aware aggregating methods. The second locality-aware implementation exposes an non-standard interface to perform additional optimization, and the authors present the additional 0.07x speedup from the extended interface. All optimizations are available in an open-source codebase, MPI Advance, which sits on top of MPI, allowing for optimizations to be added into existing codebases regardless of the system MPI install.

🛈

OSTI does not have a digital full text copy available. For more information, please see document availability, search WorldCat, or search Google Scholar.

Research Organization:: Univ. of New Mexico, Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)

DOE Contract Number:: NA0003966; CCF-2151022

OSTI ID:: 2205620

Journal Information:: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Conference: SC-W 2023: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Denver, CO (United States), 12-17 Nov 2023

Country of Publication:: United States

Language:: English

References (31)

Why is MPI (perceived to be) so complex? Holmes, Daniel J.; Skjellum, Anthony; Schafer, Derek 27th European MPI Users' Group Meeting https://doi.org/10.1145/3416315.3416318	conference	September 2020
Node-Aware Improvements to Allreduce Bienz, Amanda; Olson, Luke; Gropp, William 2019 IEEE/ACM Workshop on Exascale MPI (ExaMPI) https://doi.org/10.1109/ExaMPI49596.2019.00008	conference	November 2019
Performance Analysis and Optimal Node-aware Communication for Enlarged Conjugate Gradient Methods Lockhart, Shelby; Bienz, Amanda; Gropp, William ACM Transactions on Parallel Computing, Vol. 10, Issue 1 https://doi.org/10.1145/3580003	journal	March 2023
Detailed Modeling of Heterogeneous and Contention-Constrained Point-to-Point MPI Communication Thune, Andreas; Reinemo, Sven-Arne; Skeie, Tor IEEE Transactions on Parallel and Distributed Systems, Vol. 34, Issue 5 https://doi.org/10.1109/TPDS.2023.3253881	journal	May 2023
A Locality-Aware Bruck Allgather Bienz, Amanda; Gautam, Shreeman; Kharel, Amun Proceedings of the 29th European MPI Users' Group Meeting https://doi.org/10.1145/3555819.3555825	conference	September 2022
Communication optimization strategies for distributed deep neural network training: A survey Ouyang, Shuo; Dong, Dezun; Xu, Yemao Journal of Parallel and Distributed Computing, Vol. 149 https://doi.org/10.1016/j.jpdc.2020.11.005	journal	March 2021
Node aware sparse matrix–vector multiplication Bienz, Amanda; Gropp, William D.; Olson, Luke N. Journal of Parallel and Distributed Computing, Vol. 130 https://doi.org/10.1016/j.jpdc.2019.03.016	journal	August 2019
Designing multi-leader-based Allgather algorithms for multi-core clusters Kandalla, Krishna; Subramoni, Hari; Santhanaraman, Gopal 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5160896	conference	May 2009
Petascale XCT: 3D Image Reconstruction with Hierarchical Communications on Multi-GPU Nodes Hidayetoglu, Mert; Bicer, Tekin; de Gonzalo, Simon Garcia SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00041	conference	November 2020
Improving Performance Models for Irregular Point-to-Point Communication Bienz, Amanda; Gropp, William D.; Olson, Luke N. EuroMPI'18: 25th European MPI Users' Group Meeting, Proceedings of the 25th European MPI Users' Group Meeting https://doi.org/10.1145/3236367.3236368	conference	September 2018
Looking under the hood of the IBM Blue Gene/Q network Chen, Dong; Eisley, Noel; Heidelberger, Philip 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.72	conference	November 2012
BoomerAMG: A parallel algebraic multigrid solver and preconditioner Henson, Van Emden; Yang, Ulrike Meier Applied Numerical Mathematics, Vol. 41, Issue 1 https://doi.org/10.1016/S0168-9274(01)00115-5	journal	April 2002
Efficient Allgather for Regular SMP-Clusters Träff, Jesper Larsson Recent Advances in Parallel Virtual Machine and Message Passing Interface https://doi.org/10.1007/11846802_16	book	January 2006
The Exascale Era is Upon Us: The Frontier supercomputer may be the first to reach 1,000,000,000,000,000,000 operations per second Schneider, David IEEE Spectrum, Vol. 59, Issue 1 https://doi.org/10.1109/MSPEC.2022.9676353	journal	January 2022
Reducing communication in algebraic multigrid with multi-step node aware communication Bienz, Amanda; Gropp, William D.; Olson, Luke N. The International Journal of High Performance Computing Applications, Vol. 34, Issue 5 https://doi.org/10.1177/1094342020925535	journal	June 2020
Modeling MPI Communication Performance on SMP Nodes: Is it Time to Retire the Ping Pong Test Gropp, William; Olson, Luke N.; Samfass, Philipp Proceedings of the 23rd European MPI Users' Group Meeting on - EuroMPI 2016 https://doi.org/10.1145/2966884.2966919	conference	January 2016
A Hierarchical and Load-Aware Design for Large Message Neighborhood Collectives Ghazimirsaeed, S. Mahdieh; Zhou, Qinghua; Ruhela, Amit SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00038	conference	November 2020
Implementation and evaluation of MPI 4.0 partitioned communication libraries Dosanjh, Matthew G. F.; Worley, Andrew; Schafer, Derek Parallel Computing, Vol. 108 https://doi.org/10.1016/j.parco.2021.102827	journal	December 2021
Overview of the Blue Gene/L system architecture Gara, A.; Blumrich, M. A.; Chen, D. IBM Journal of Research and Development, Vol. 49, Issue 2.3 https://doi.org/10.1147/rd.492.0195	journal	March 2005
Optimization of MPI persistent communication Hatanaka, Masayuki; Hori, Atsushi; Ishikawa, Yutaka Proceedings of the 20th European MPI Users' Group Meeting https://doi.org/10.1145/2488551.2488566	conference	September 2013
HierKNEM: An Adaptive Framework for Kernel-Assisted and Topology-Aware Collective Communications on Many-core Clusters Ma, Teng; Bosilca, George; Bouteiller, Aurelien 2012 IEEE 26th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2012.91	conference	May 2012
Characterizing the performance of node-aware strategies for irregular point-to-point communication on heterogeneous architectures Lockhart, Shelby; Bienz, Amanda; Gropp, William D. Parallel Computing, Vol. 116 https://doi.org/10.1016/j.parco.2023.103021	journal	July 2023
Stepping up to Summit Hines, Jonathan Computing in Science & Engineering, Vol. 20, Issue 2 https://doi.org/10.1109/MCSE.2018.021651341	journal	March 2018
Designing broadcasting algorithms in the postal model for message-passing systems Bar-Noy, Amotz; Kipnis, Shlomo Proceedings of the fourth annual ACM symposium on Parallel algorithms and architectures - SPAA '92 https://doi.org/10.1145/140901.140903	conference	January 1992
Modeling Data Movement Performance on Heterogeneous Architectures Bienz, Amanda; Olson, Luke N.; Gropp, William D. 2021 IEEE High Performance Extreme Computing Conference (HPEC) https://doi.org/10.1109/HPEC49654.2021.9622742	conference	September 2021
Cheetah: A Framework for Scalable Hierarchical Collective Operations Graham, Richard; Venkata, Manjunath Gorentla; Ladd, Joshua 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2011.42	conference	May 2011
Exploiting hierarchy in parallel computer networks to optimize collective operation performance Karonis, N. T.; de Supinski, B. R.; Foster, I. Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000 https://doi.org/10.1109/IPDPS.2000.846009	conference	January 2000
Topology-Aware Rank Reordering for MPI Collectives Mirsadeghi, Seyed H.; Afsahi, Ahmad 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW.2016.139	conference	May 2016
Bandwidth Efficient All-reduce Operation on Tree Topologies Patarasuk, Pitch; Yuan, Xin 2007 IEEE International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2007.370405	conference	March 2007
PETSc (Portable, Extensible Toolkit for Scientific Computation) Leasure, Bruce; Kuck, David J.; Gorlatch, Sergei Encyclopedia of Parallel Computing https://doi.org/10.1007/978-0-387-09766-4_87	book	January 2011
Decomposing MPI Collectives for Exploiting Multi-lane Communication Traff, Jesper Larsson; Hunold, Sascha 2020 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER49012.2020.00037	conference	September 2020

Similar Records

Exploring MPI Communication Models for Graph Applications Using Graph Matching as a Case Study

Conference · 2019 · OSTI ID:1572673

Ghosh, Sayan; Halappanavar, Mahantesh; Kalyanaraman, Anantharaman; +2 more

Deploy threading in Nalu solver stack

Technical Report · 2018 · OSTI ID:1481562

Prokopenko, Andrey; Thomas, Stephen; Swirydowicz, Kasia; +4 more

Quantum Monte Carlo Endstation for Petascale Computing

Technical Report · 2011 · OSTI ID:1007216

Ceperley, David

Related Subjects

Hypre
AMG
MPI
neighborhood collectives
localityaware parallelism
persistent communication

Optimizing Irregular Communication with Neighborhood Collectives and Locality-Aware Parallelism

Citation Formats

References (31)

Similar Records

Related Subjects