The PetscSF Scalable Communication Layer

Journal Article · Wed May 26 00:00:00 EDT 2021 · IEEE Transactions on Parallel and Distributed Systems

DOI:https://doi.org/10.1109/TPDS.2021.3084070· OSTI ID:1837203

Zhang, Junchao ^[1]; Brown, Jed ^[2]; Balay, Satish ^[1]; Faibussowitsch, Jacob ^[3]; Knepley, Matthew ^[4]; Marin, Oana ^[1]; Mills, Richard Tran ^[1]; Munson, Todd ^[1]; Smith, Barry F. ^[1]; Zampini, Stefano ^[5]

---

1. Argonne National Lab. (ANL), Argonne, IL (United States)
2. Univ. of Colorado, Boulder, CO (United States)
3. Univ. of Illinois at Urbana-Champaign, IL (United States)
4. Univ. at Buffalo, NY (United States)
5. King Abdullah University of Science and Technology (KAUST), Thuwal (Saudi Arabia)

PetscSF, the communication component of the Portable, Extensible Toolkit for Scientific Computation (PETSc), is designed to provide PETSc's communication infrastructure suitable for exascale computers that utilize GPUs and other accelerators. PetscSF provides a simple application programming interface (API) for managing common communication patterns in scientific computations by using a star-forest graph representation. PetscSF supports several implementations based on MPI and NVSHMEM, whose selection is based on the characteristics of the application or the target architecture. An efficient and portable model for network and intra-node communication is essential for implementing large-scale applications. The Message Passing Interface, which has been the de facto standard for distributed memory systems, has developed into a large complex API that does not yet provide high performance on the emerging heterogeneous CPU-GPU-based exascale systems. Here, we discuss the design of PetscSF, how it can overcome some difficulties of working directly with MPI on GPUs, and we demonstrate its performance, scalability, and novel features.

• Research Organization: Argonne National Laboratory (ANL), Argonne, IL (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC)
• Grant/Contract Number: AC02-06CH11357; AC05-00OR22725; SC0016140
• OSTI ID: 1837203
• Journal Information: IEEE Transactions on Parallel and Distributed Systems, Journal Name: IEEE Transactions on Parallel and Distributed Systems Journal Issue: 4 Vol. 33; ISSN 1045-9219
• Publisher: IEEE
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Communication
GPU
MPI
PETSc
extreme-scale