Network-friendly one-sided communication through multinode cooperation on petascale cray xt5 systems

Conference · Sat Jan 01 00:00:00 EST 2011

OSTI ID:1009956

Tipparaju, Vinod ^[1]; Que, Xinyu ^[1]; Yu, Weikuan ^[1]; Vetter, Jeffrey S ^[1]

---

1. ORNL

ne-sided communication is important to enable asynchronous communication and data movement for Global Address Space (GAS) programming models. Such communication is typically realized through direct messages between initiator and target processes. For petascale systems with 10,000s of nodes and 100,000s of cores, these direct messages require dedicated communication buffers and/or channels, which can lead to significant scalability challenges for GAS programming models. In this paper, we describe a network-friendly communication model, multinode cooperation, to enable indirect one-sided communication. Compute nodes work together to handle one-sided requests through (1) request forwarding in which one node can intercept a request and forward it to a target node, and (2) request aggregation in which one node can aggregate many requests to a target node. We have implemented multinode cooperation for a popular GAS runtime library, Aggregate Remote Memory Copy Interface (ARMCI). Our experimental results on a large-scale Cray XT5 system demonstrate that, multinode cooperation is able to greatly increase the memory scalability by reducing the number of communication buffers. In addition, multinode cooperation improves the resiliency of GAS runtime system to network contention. Furthermore, multinode cooperation can benefit the performance of scientific applications. In one case, it reduces the total execution time of an NWChem application by 52%.

• Research Organization: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). National Center for Computational Sciences (NCCS)
• Sponsoring Organization: USDOE Office of Science (SC)
• DOE Contract Number: DE-AC05-00OR22725
• OSTI ID: 1009956
• Resource Relation: Conference: IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, Newport Beach, CA, USA, 20110523, 20110523
• Country of Publication: United States
• Language: English