Designing and prototyping extensions to the Message Passing Interface in MPICH

Zhou, Hui; Raffenetti, Ken; Guo, Yanfei; Gillis, Thomas; Latham, Robert; Thakur, Rajeev

doi:10.1177/10943420241263544

Designing and prototyping extensions to the Message Passing Interface in MPICH

Journal Article · Mon Aug 19 00:00:00 EDT 2024 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/10943420241263544· OSTI ID:2571429

^[1]; ^[1]; ^[1]; ^[1]; ^[1]; ^[1]

Argonne National Laboratory (ANL), Argonne, IL (United States)

As HPC system architectures and the applications running on them continue to evolve, the MPI standard itself must evolve. The trend in current and future HPC systems toward powerful nodes with multiple CPU cores and multiple GPU accelerators makes efficient support for hybrid programming critical for applications to achieve high performance. However, the support for hybrid programming in the MPI standard has not kept up with recent trends. The MPICH implementation of MPI provides a platform for implementing and experimenting with new proposals and extensions to fill this gap and to gain valuable experience and feedback before the MPI Forum can consider them for standardization. Here, in this work, we detail six extensions implemented in MPICH to increase MPI interoperability with other runtimes, with a specific focus on heterogeneous architectures. First, the extension to MPI generalized requests lets applications integrate asynchronous tasks into MPI’s progress engine. Second, the iovec extension to datatypes lets applications use MPI datatypes as a general-purpose data layout API beyond just MPI communications. Third, a new MPI object, MPIX_Stream, can be used by applications to identify execution contexts beyond MPI processes, including threads and GPU streams. MPIX stream communicators can be created to make existing MPI functions thread-aware and GPU-aware, thus providing applications with explicit ways to achieve higher performance. Fourth, MPIX Streams are extended to support the enqueue semantics for offloading MPI communications onto a GPU stream context. Fifth, thread communicators allow MPI communicators to be constructed with individual threads, thus providing a new level of interoperability between MPI and on-node runtimes such as OpenMP. Lastly, we present an extension to invoke MPI progress, which lets users spawn progress threads with fine-grained control to adapt the communication performance to their application designs. We describe the design and implementation of these extensions, provide usage examples, and highlight their expected benefits with performance results.

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC)

Grant/Contract Number:: AC02-06CH11357

OSTI ID:: 2571429

Journal Information:: International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Vol. 38; ISSN 1094-3420; ISSN 1741-2846

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

References (13)

A survey of MPI usage in the US exascale computing project: A survey of MPI usage in the U. S. exascale computing project Bernholdt, David E.; Boehm, Swen; Bosilca, George Concurrency and Computation: Practice and Experience https://doi.org/10.1002/cpe.4851	journal	September 2018
The design and evolution of Zipcode Skjellum, Anthony; Smith, Steven G.; Doss, Nathan E. Parallel Computing, Vol. 20, Issue 4 https://doi.org/10.1016/0167-8191(94)90029-9	journal	April 1994
Portable programming with the PARMACS message-passing library Calkin, R.; Hempel, R.; Hoppe, H. -C Parallel Computing, Vol. 20, Issue 4 https://doi.org/10.1016/0167-8191(94)90031-0	journal	April 1994
Performance models for asynchronous data transfers on consumer Graphics Processing Units Gómez-Luna, Juan; González-Linares, José María; Benavides, José Ignacio Journal of Parallel and Distributed Computing, Vol. 72, Issue 9 https://doi.org/10.1016/j.jpdc.2011.07.011	journal	September 2012
Performance and Power Efficient Massive Parallel Computational Model for HPC Heterogeneous Exascale Systems Ashraf, M. Usman; Alburaei Eassa, Fathy; Ahmad Albeshri, Aiiad IEEE Access, Vol. 6 https://doi.org/10.1109/ACCESS.2018.2823299	journal	January 2018
Design and evaluation of Nemesis, a scalable, low-latency, message-passing communication subsystem Buntinas, D.; Mercier, G.; Gropp, W. Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06) https://doi.org/10.1109/CCGRID.2006.31	conference	January 2006
Give MPI Threading a Fair Chance: A Study of Multithreaded MPI Designs Patinyasakdikul, Thananon; Eberius, David; Bosilca, George 2019 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2019.8891015	conference	September 2019
Logically Parallel Communication for Fast MPI+Threads Applications Zambre, Rohit; Sahasrabudhe, Damodar; Zhou, Hui IEEE Transactions on Parallel and Distributed Systems, Vol. 32, Issue 12 https://doi.org/10.1109/TPDS.2021.3075157	journal	December 2021
MPI+Threads: runtime contention and remedies Amer, Abdelhalim; Lu, Huiwei; Wei, Yanjie ACM SIGPLAN Notices, Vol. 50, Issue 8 https://doi.org/10.1145/2858788.2688522	journal	January 2015
Multi-criteria partitioning of multi-block structured grids Wang, Hengjie; Chandramowlishwaran, Aparna Proceedings of the ACM International Conference on Supercomputing https://doi.org/10.1145/3330345.3330369	conference	June 2019
How I learned to stop worrying about user-visible endpoints and love MPI Zambre, Rohit; Chandramowliswharan, Aparna; Balaji, Pavan Proceedings of the 34th ACM International Conference on Supercomputing https://doi.org/10.1145/3392717.3392773	conference	June 2020
MPIX Stream: An Explicit Solution to Hybrid MPI+X Programming Zhou, Hui; Raffenetti, Ken; Guo, Yanfei Proceedings of the 29th European MPI Users' Group Meeting https://doi.org/10.1145/3555819.3555820	conference	September 2022
Performance comparison of MPI and three openMP programming styles on shared memory multiprocessors Krawezik, Géraud Proceedings of the fifteenth annual ACM symposium on Parallel algorithms and architectures - SPAA '03 https://doi.org/10.1145/777412.777433	conference	January 2003