Cheetah: A Framework for Scalable Hierarchical Collective Operations

Conference · Sat Jan 01 04:00:00 EST 2011

OSTI ID:1035530

Graham, Richard L ^[1]; Gorentla Venkata, Manjunath ^[1]; Ladd, Joshua S ^[1]; Shamis, Pavel ^[1]; Rabinovitz, Ishai ^[2]; Filipov, Vasily ^[2]; Shainer, Gilad ^[2]

---

1. ORNL
2. Mellanox Technologies, Inc.

Collective communication operations, used by many scientific applications, tend to limit overall parallel application performance and scalability. Computer systems are becoming more heterogeneous with increasing node and core-per-node counts. Also, a growing number of data-access mechanisms, of varying characteristics, are supported within a single computer system. We describe a new hierarchical collective communication framework that takes advantage of hardware-specific data-access mechanisms. It is flexible, with run-time hierarchy specification, and sharing of collective communication primitives between collective algorithms. Data buffers are shared between levels in the hierarchy reducing collective communication management overhead. We have implemented several versions of the Message Passing Interface (MPI) collective operations, MPI Barrier() and MPI Bcast(), and run experiments using up to 49, 152 processes on a Cray XT5, and a small InfiniBand based cluster. At 49, 152 processes our barrier implementation outperforms the optimized native implementation by 75%. 32 Byte and one Mega-Byte broadcasts outperform it by 62% and 11%, respectively, with better scalability characteristics. Improvements relative to the default Open MPI implementation are much larger.

• Research Organization: Oak Ridge National Laboratory (ORNL)
• Sponsoring Organization: SC USDOE - Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1035530
• Country of Publication: United States
• Language: English

Related Subjects

99 GENERAL AND MISCELLANEOUS
ALGORITHMS
BUFFERS
CLOUDS
COMMUNICATIONS
COMPUTERS
IMPLEMENTATION
MANAGEMENT
PERFORMANCE