Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement

Conference · Wed Jan 01 00:00:00 EST 2014

OSTI ID:1185602

Wang, Feiyi ^[1]; Oral, H Sarp ^[1]; Gupta, Saurabh ^[1]; Tiwari, Devesh ^[1]; Vazhkudai, Sudharshan S ^[1]

---

1. ORNL

With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus, with more applications putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability and scalability requirements of data intensive applications. The sharing nature of storage resources and the intrinsic interactions across these layers make it to realize user-level, end-to-end performance gains a great challenge. We propose a topology-aware resource load balancing strategy to improve per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both the reduction of application run times and higher resolution simulation runs.

• Research Organization: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
• Sponsoring Organization: USDOE Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1185602
• Resource Relation: Conference: The 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS 2014), Hsinchu, Taiwan, 20141216, 20141219
• Country of Publication: United States
• Language: English