ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications

Conference · Sat Jan 01 04:00:00 EST 2011

OSTI ID:1014252

Gorentla Venkata, Manjunath ^[1]; Graham, Richard L ^[1]; Ladd, Joshua S ^[1]; Shamis, Pavel ^[1]; Rabinovitz, Ishai ^[2]; Filipov, Vasily ^[2]; Shainer, Gilad ^[2]

---

1. ORNL
2. Mellanox Technologies, Inc.

This paper describes the design and implementation of InfiniBand (IB) CORE-Direct based blocking and nonblocking broadcast operations within the Cheetah collective operation framework. It describes a novel approach that fully ofFLoads collective operations and employs only user-supplied buffers. For a 64 rank communicator, the latency of CORE-Direct based hierarchical algorithm is better than production-grade Message Passing Interface (MPI) implementations, 150% better than the default Open MPI algorithm and 115% better than the shared memory optimized MVAPICH implementation for a one kilobyte (KB) message, and for eight mega-bytes (MB) it is 48% and 64% better, respectively. Flat-topology broadcast achieves 99.9% overlap in a polling based communication-computation test, and 95.1% overlap for a wait based test, compared with 92.4% and 17.0%, respectively, for a similar Central Processing Unit (CPU) based implementation.

• Research Organization: Oak Ridge National Laboratory (ORNL)
• Sponsoring Organization: SC USDOE - Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1014252
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ALGORITHMS
BUFFERS
COMMUNICATIONS
DESIGN
IMPLEMENTATION
PROCESSING