skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement

Abstract

With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus, with more applications putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability and scalability requirements of data intensive applications. The sharing nature of storage resources and the intrinsic interactions across these layers make it to realize user-level, end-to-end performance gains a great challenge. We propose a topology-aware resource load balancing strategy to improve per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both the reduction of application run times and higher resolution simulation runs.

Authors:
 [1];  [1];  [1];  [1];  [1]
  1. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1185602
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: The 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS 2014), Hsinchu, Taiwan, 20141216, 20141219
Country of Publication:
United States
Language:
English

Citation Formats

Wang, Feiyi, Oral, H Sarp, Gupta, Saurabh, Tiwari, Devesh, and Vazhkudai, Sudharshan S. Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement. United States: N. p., 2014. Web.
Wang, Feiyi, Oral, H Sarp, Gupta, Saurabh, Tiwari, Devesh, & Vazhkudai, Sudharshan S. Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement. United States.
Wang, Feiyi, Oral, H Sarp, Gupta, Saurabh, Tiwari, Devesh, and Vazhkudai, Sudharshan S. Wed . "Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement". United States. https://www.osti.gov/servlets/purl/1185602.
@article{osti_1185602,
title = {Improving Large-scale Storage System Performance via Topology-aware and Balanced Data Placement},
author = {Wang, Feiyi and Oral, H Sarp and Gupta, Saurabh and Tiwari, Devesh and Vazhkudai, Sudharshan S},
abstractNote = {With the advent of big data, the I/O subsystems of large-scale compute clusters are becoming a center of focus, with more applications putting greater demands on end-to-end I/O performance. These subsystems are often complex in design. They comprise of multiple hardware and software layers to cope with the increasing capacity, capability and scalability requirements of data intensive applications. The sharing nature of storage resources and the intrinsic interactions across these layers make it to realize user-level, end-to-end performance gains a great challenge. We propose a topology-aware resource load balancing strategy to improve per-application I/O performance. We demonstrate the effectiveness of our algorithm on an extreme-scale compute cluster, Titan, at the Oak Ridge Leadership Computing Facility (OLCF). Our experiments with both synthetic benchmarks and a real-world application show that, even under congestion, our proposed algorithm can improve large-scale application I/O performance significantly, resulting in both the reduction of application run times and higher resolution simulation runs.},
doi = {},
url = {https://www.osti.gov/biblio/1185602}, journal = {},
number = ,
volume = ,
place = {United States},
year = {2014},
month = {1}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: