ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications
- ORNL
- Mellanox Technologies, Inc.
This paper describes the design and implementation of InfiniBand (IB) CORE-Direct based blocking and nonblocking broadcast operations within the Cheetah collective operation framework. It describes a novel approach that fully ofFLoads collective operations and employs only user-supplied buffers. For a 64 rank communicator, the latency of CORE-Direct based hierarchical algorithm is better than production-grade Message Passing Interface (MPI) implementations, 150% better than the default Open MPI algorithm and 115% better than the shared memory optimized MVAPICH implementation for a one kilobyte (KB) message, and for eight mega-bytes (MB) it is 48% and 64% better, respectively. Flat-topology broadcast achieves 99.9% overlap in a polling based communication-computation test, and 95.1% overlap for a wait based test, compared with 92.4% and 17.0%, respectively, for a similar Central Processing Unit (CPU) based implementation.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- DE-AC05-00OR22725
- OSTI ID:
- 1014252
- Resource Relation:
- Conference: International Parallel and Distributed Processing Symposium, CASS Workshop, Anchorage, AK, USA, 20110516, 20110516
- Country of Publication:
- United States
- Language:
- English
Similar Records
Overlapping Computation and Communication: Barrier Algorithms and ConnectX-2 CORE-Direct Capabilities
Overlapping Computation and Communication: Barrier Algorithms and ConnectX-2 CORE-Direct Capabilities