Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA

Spies, Lukas; Bienz, Amanda; Moulton, John David; Olson, Luke; Reisner, Andrew Ray

doi:10.1016/j.parco.2022.102973

Title: Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA

Abstract

Exchanging halo data is a common task in modern scientific computing applications and efficient handling of this operation is critical for the performance of the overall simulation. Tausch is a novel header-only library that provides a simple API for efficiently handling these types of data movements. Tausch supports both simple CPU-only systems, but also more complex heterogeneous systems with both CPUs and GPUs. It currently supports both OpenCL and CUDA for communicating with GPGPU devices, and allows for communication between GPGPUs and CPUs. The API allows for drop-in replacement in existing codes and can be used for the communication layer in new codes. This paper provides an overview of the approach taken in Tausch, and a performance analysis that demonstrates expected and achieved performance. Here, we highlight the ease of use and performance with three applications: First Tausch is compared to the halo exchange framework from two Mantevo applications, HPCCG and miniFE, and then it is used to replace a legacy halo exchange library in the flexible multigrid solver framework Cedar.

Authors:

^[1];

^[2];

^[3];

^[1];

^[3]

Univ. of Illinois at Urbana-Champaign, IL (United States)
Univ. of New Mexico, Albuquerque, NM (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Publication Date:: Fri Sep 23 00:00:00 EDT 2022

Research Org.:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)

OSTI Identifier:: 1890986

Alternate Identifier(s):: OSTI ID: 1890549

Report Number(s):: LA-UR-21-28891
Journal ID: ISSN 0167-8191

Grant/Contract Number:: 89233218CNA000001; NA0002374; OCI-0725070; ACI-1238993

Resource Type:: Accepted Manuscript

Journal Name:: Parallel Computing

Additional Journal Information:: Journal Volume: 114; Journal ID: ISSN 0167-8191

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Halo; Exchange; Tausch; Mpi; Opencl; Cuda; C++; Heterogeneous; Performance

Citation Formats


                    Spies, Lukas, Bienz, Amanda, Moulton, John David, Olson, Luke, and Reisner, Andrew Ray. Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA.  United States: N. p., 2022. 
Web.  doi:10.1016/j.parco.2022.102973.

Copy to clipboard


                    Spies, Lukas, Bienz, Amanda, Moulton, John David, Olson, Luke, & Reisner, Andrew Ray. Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA.  United States.  https://doi.org/10.1016/j.parco.2022.102973

Copy to clipboard


                    Spies, Lukas, Bienz, Amanda, Moulton, John David, Olson, Luke, and Reisner, Andrew Ray. Fri .  
"Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA".  United States.  https://doi.org/10.1016/j.parco.2022.102973.  https://www.osti.gov/servlets/purl/1890986.

Copy to clipboard


                    
@article{osti_1890986,

  title        = {Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA},

  author       = {Spies, Lukas and Bienz, Amanda and Moulton, John David and Olson, Luke and Reisner, Andrew Ray},

  abstractNote = {Exchanging halo data is a common task in modern scientific computing applications and efficient handling of this operation is critical for the performance of the overall simulation. Tausch is a novel header-only library that provides a simple API for efficiently handling these types of data movements. Tausch supports both simple CPU-only systems, but also more complex heterogeneous systems with both CPUs and GPUs. It currently supports both OpenCL and CUDA for communicating with GPGPU devices, and allows for communication between GPGPUs and CPUs. The API allows for drop-in replacement in existing codes and can be used for the communication layer in new codes. This paper provides an overview of the approach taken in Tausch, and a performance analysis that demonstrates expected and achieved performance. Here, we highlight the ease of use and performance with three applications: First Tausch is compared to the halo exchange framework from two Mantevo applications, HPCCG and miniFE, and then it is used to replace a legacy halo exchange library in the flexible multigrid solver framework Cedar.},

  doi          = {10.1016/j.parco.2022.102973},

  journal      = {Parallel Computing},

  number       = ,

  volume       = 114,

  place        = {United States},

  year         = {Fri Sep 23 00:00:00 EDT 2022},

  month        = {Fri Sep 23 00:00:00 EDT 2022}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.parco.2022.102973

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

An overview of the Trilinos project
journal, September 2005

Heroux, Michael A.; Phipps, Eric T.; Salinger, Andrew G.
ACM Transactions on Mathematical Software, Vol. 31, Issue 3
DOI: 10.1145/1089014.1089021

Kokkos: Enabling manycore performance portability through polymorphic memory access patterns
journal, December 2014

Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel
Journal of Parallel and Distributed Computing, Vol. 74, Issue 12
DOI: 10.1016/j.jpdc.2014.07.003

Scalable line and plane relaxation in a parallel structured multigrid solver
journal, December 2020

Reisner, Andrew; Berndt, Markus; Moulton, J. David
Parallel Computing, Vol. 100
DOI: 10.1016/j.parco.2020.102705

Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution
journal, January 2018

Reisner, Andrew; Olson, Luke N.; Moulton, J. David
SIAM Journal on Scientific Computing, Vol. 40, Issue 4
DOI: 10.1137/17M1146440

Similar Records in DOE PAGES and OSTI.GOV collections:

Performance Analysis of a High-Level Abstractions-Based Hydrocode on Future Computing Systems

Journal Article

In this paper we present research on applying a domain specific high-level abstractions (HLA) development strategy with the aim to “future-proof” a key class of high performance computing (HPC) applications that simulate hydrodynamics computations at AWE plc. We build on an existing high-level abstraction framework, OPS, that is being developed for the solution of multi-block structured mesh-based applications at the University of Oxford. OPS uses an “active library” approach where a single application code written using the OPS API can be transformed into different highly optimized parallel implementations which can then be linked against the appropriate parallel library enabling executionmore »« less
https://doi.org/10.1007/978-3-319-17248-4_5
A Distributed OpenCL Framework using Redundant Computation and Data Replication

Conference Kim, Junghyun ; Gangwon, Jo ; Jaehoon, Jung ; ...

Applications written solely in OpenCL or CUDA cannot execute on a cluster as a whole. Most previous approaches that extend these programming models to clusters are based on a common idea: designating a centralized host node and coordinating the other nodes with the host for computation. However, the centralized host node is a serious performance bottleneck when the number of nodes is large. In this paper, we propose a scalable and distributed OpenCL framework called SnuCL-D for large-scale clusters. SnuCL-D's remote device virtualization provides an OpenCL application with an illusion that all compute devices in a cluster are confined inmore »« less
Full Text Available
Implementing Molecular Dynamics for Hybrid High Performance Computers - 1. Short Range Forces

Journal Article Brown, W Michael ; Wang, Peng ; Plimpton, Steven J ; ... - Computer Physics Communications

The use of accelerators such as general-purpose graphics processing units (GPGPUs) have become popular in scientific computing applications due to their low cost, impressive floating-point capabilities, high memory bandwidth, and low electrical power requirements. Hybrid high performance computers, machines with more than one type of floating-point processor, are now becoming more prevalent due to these advantages. In this work, we discuss several important issues in porting a large molecular dynamics code for use on parallel hybrid machines - 1) choosing a hybrid parallel decomposition that works on central processing units (CPUs) with distributed memory and accelerator cores with shared memory,more »« less
https://doi.org/10.1016/j.cpc.2010.12.021
Heterogeneous computing with OpenMP and Hydra

Journal Article Diener, Matthias ; Kale, Laxmikant V. ; Bodony, Daniel J. - Concurrency and Computation. Practice and Experience

Summary High‐performance computing relies on accelerators (such as GPGPUs) to achieve fast execution of scientific applications. Traditionally, these accelerators have been programmed with specialized languages, such as CUDA or OpenCL. In recent years, OpenMP emerged as a promising alternative for supporting accelerators, providing advantages such as maintaining a single code base for the host and different accelerator types and providing a simple way to extend support for accelerators to existing application codes. Efficiently using this support requires solving several challenges, related to performance, work partitioning, and concurrent execution on multiple device types. In this article, we discuss our experiences withmore »« less
Cited by 2
https://doi.org/10.1002/cpe.5728
High performance graphics processor based computed tomography reconstruction algorithms for nuclear and other large scale applications.

Technical Report Jimenez, Edward S. ; Orr, Laurel J. ; Thompson, Kyle R.

The goal of this work is to develop a fast computed tomography (CT) reconstruction algorithm based on graphics processing units (GPU) that achieves significant improvement over traditional central processing unit (CPU) based implementations. The main challenge in developing a CT algorithm that is capable of handling very large datasets is parallelizing the algorithm in such a way that data transfer does not hinder performance of the reconstruction algorithm. General Purpose Graphics Processing (GPGPU) is a new technology that the Science and Technology (S&T) community is starting to adopt in many fields where CPU-based computing is the norm. GPGPU programming requiresmore »« less
https://doi.org/10.2172/1121931

Full Text Available

Similar Records

Title: Tausch: A halo exchange library for large heterogeneous computing systems using MPI, OpenCL, and CUDA

Abstract

Citation Formats

An overview of the Trilinos project journal, September 2005

Kokkos: Enabling manycore performance portability through polymorphic memory access patterns journal, December 2014

Scalable line and plane relaxation in a parallel structured multigrid solver journal, December 2020

Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution journal, January 2018

An overview of the Trilinos project
journal, September 2005

Kokkos: Enabling manycore performance portability through polymorphic memory access patterns
journal, December 2014

Scalable line and plane relaxation in a parallel structured multigrid solver
journal, December 2020

Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution
journal, January 2018