End-to-End Data Movement Using MPI-IO Over Routed Terabots Infrastructures

Conference · Tue Jan 01 00:00:00 EST 2013

OSTI ID:1118788

Vallee, Geoffroy R ^[1]; Atchley, Scott ^[1]; Kim, Youngjae ^[1]; Shipman, Galen M ^[1]

---

1. ORNL

Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightly couple their domain specific applications with a post processing application. The paper presents a solution for the data transfer between MPI applications using a dedicated wide area network (WAN) terabit infrastructure. The proposed solution is based on parallel access to data files and the Message Passing Interface (MPI) over the Common Communication Infrastructure (CCI) for the data transfer over a routed infrastructure. In the context of this research, the Energy Sciences Network (ESnet) of the U.S. Department of Energy (DOE) is targeted for the transfer of data between DOE national laboratories.

• Research Organization: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC)
• DOE Contract Number: DE-AC05-00OR22725
• OSTI ID: 1118788
• Resource Relation: Conference: 3rd IEEE/ACM International Workshop on Network-aware Data Management (NDM 2013), Denver, CO, USA, 20131117, 20131117
• Country of Publication: United States
• Language: English