Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?

Huang, Dan; Liu, Qing; Choi, Jong Youl; Podhorszki, Norbert; Klasky, Scott A.; Logan, Jeremy; Ostrouchov, George; He, Xubin; Wolf, Matthew D.

doi:10.1109/TC.2018.2881709

Title: Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?

Full Record
Other Related Research

Abstract

For a production high-performance computing (HPC) system, where storage devices are shared between multiple applications and managed in a best effort manner, I/O contention is often a major problem. In this paper, we propose a balanced messaging-based re-routing in conjunction with throttling at the middleware level. This work tackles two key challenges that have not been fully resolved in the past: whether I/O variability can be reduced on a QoS-less HPC storage system, and how to design a runtime scheduling system that can scale up to a large amount of cores. The proposed scheme uses a two-level messaging system to re-route I/O requests to a less congested storage location so that write performance is improved, while limiting the impact on read by throttling re-routing. An analytical model is derived to guide the setup of optimal throttling factor. We thoroughly analyze the virtual messaging layer overhead and explore whether the in-transit buffering is effective in managing I/O variability. Contrary to the intuition, in-transit buffer cannot completely solve the problem. It can reduce the absolute variability but not the relative variability. Here, the proposed scheme is verified against a synthetic benchmark as well as being used by production applications.

Authors:

^[1];

^[2];

^[3];

^[2]

New Jersey Inst. of Technology, Newark, NJ (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Temple Univ., Philadelphia, PA (United States)

Publication Date:: Mon Nov 19 00:00:00 EST 2018

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1559601

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Transactions on Computers

Additional Journal Information:: Journal Volume: 68; Journal Issue: 5; Journal ID: ISSN 0018-9340

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; High-performance computing; storage; quality of service; variability

Citation Formats


                    Huang, Dan, Liu, Qing, Choi, Jong Youl, Podhorszki, Norbert, Klasky, Scott A., Logan, Jeremy, Ostrouchov, George, He, Xubin, and Wolf, Matthew D. Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?.  United States: N. p., 2018. 
Web.  doi:10.1109/TC.2018.2881709.

Copy to clipboard


                    Huang, Dan, Liu, Qing, Choi, Jong Youl, Podhorszki, Norbert, Klasky, Scott A., Logan, Jeremy, Ostrouchov, George, He, Xubin, & Wolf, Matthew D. Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?.  United States.  https://doi.org/10.1109/TC.2018.2881709

Copy to clipboard


                    Huang, Dan, Liu, Qing, Choi, Jong Youl, Podhorszki, Norbert, Klasky, Scott A., Logan, Jeremy, Ostrouchov, George, He, Xubin, and Wolf, Matthew D. Mon .  
"Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?".  United States.  https://doi.org/10.1109/TC.2018.2881709.  https://www.osti.gov/servlets/purl/1559601.

Copy to clipboard


                    
@article{osti_1559601,

  title        = {Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?},

  author       = {Huang, Dan and Liu, Qing and Choi, Jong Youl and Podhorszki, Norbert and Klasky, Scott A. and Logan, Jeremy and Ostrouchov, George and He, Xubin and Wolf, Matthew D.},

  abstractNote = {For a production high-performance computing (HPC) system, where storage devices are shared between multiple applications and managed in a best effort manner, I/O contention is often a major problem. In this paper, we propose a balanced messaging-based re-routing in conjunction with throttling at the middleware level. This work tackles two key challenges that have not been fully resolved in the past: whether I/O variability can be reduced on a QoS-less HPC storage system, and how to design a runtime scheduling system that can scale up to a large amount of cores. The proposed scheme uses a two-level messaging system to re-route I/O requests to a less congested storage location so that write performance is improved, while limiting the impact on read by throttling re-routing. An analytical model is derived to guide the setup of optimal throttling factor. We thoroughly analyze the virtual messaging layer overhead and explore whether the in-transit buffering is effective in managing I/O variability. Contrary to the intuition, in-transit buffer cannot completely solve the problem. It can reduce the absolute variability but not the relative variability. Here, the proposed scheme is verified against a synthetic benchmark as well as being used by production applications.},

  doi          = {10.1109/TC.2018.2881709},

  journal      = {IEEE Transactions on Computers},

  number       = 5,

  volume       = 68,

  place        = {United States},

  year         = {Mon Nov 19 00:00:00 EST 2018},

  month        = {Mon Nov 19 00:00:00 EST 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TC.2018.2881709

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Can I/O Variability Be Reduced on QoS-Less HPC Storage Systems?

Journal Article Huang, Dan ; Liu, Qing ; Choi, Jong ; ... - IEEE Transactions on Computers

For a production high-performance computing (HPC) system, where storage devices are shared between multiple applications and managed in a best effort manner, I/O contention is often a major problem. In this paper, we propose a balanced messaging-based re-routing in conjunction with throttling at the middleware level. This work tackles two key challenges that have not been fully resolved in the past: whether I/O variability can be reduced on a QoS-less HPC storage system, and how to design a runtime scheduling system that can scale up to a large amount of cores. The proposed scheme uses a two-level messaging system tomore »« less
https://doi.org/10.1109/TC.2018.2881709
StoreRush: An Application-Level Approach to Harvesting Idle Storage in a Best Effort Environment

Journal Article Liu, Qing ; Podhorszki, Norbert ; Choi, Jong ; ... - Procedia Computer Science

For a production HPC system where storage devices are shared between multiple applications and managed in a best effort manner, contention is often a major problem leading to some storage devices being more loaded than others and causing a significant reduction in I/O throughput. In this paper, we describe our latest efforts StoreRush to resolve this practical issue at the application level without requiring modification to the file and storage system. The proposed scheme uses a two-level messaging system to harvest idle storage via re-routing I/O requests to a less congested storage location so that write performance is improved whilemore »« less
Cited by 1
https://doi.org/10.1016/j.procs.2017.05.005

Full Text Available
Institute for Sustained Performance, Energy, and Resilience (SuPER)

Technical Report Jagode, Heike ; Bosilca, George ; Danalis, Anthony ; ...

The University of Tennessee (UTK) and University of Texas at El Paso (UTEP) partnership supported the three main thrusts of the SUPER project---performance, energy, and resilience. The UTK-UTEP effort thus helped advance the main goal of SUPER, which was to ensure that DOE's computational scientists can successfully exploit the emerging generation of high performance computing (HPC) systems. This goal is being met by providing application scientists with strategies and tools to productively maximize performance, conserve energy, and attain resilience. The primary vehicle through which UTK provided performance measurement support to SUPER and the larger HPC community is the Performance Applicationmore »« less
https://doi.org/10.2172/1333889

Full Text Available
...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

Conference Klasky, Scott A ; Lofstead, J. ; Bent, John ; ...

As HPC applications run on increasingly high process counts on larger and larger machines, both the frequency of checkpoints needed for fault tolerance and the resolution and size of Data Analysis Dumps are expected to increase proportionally. In order to maintain an acceptable ratio of time spent performing useful computation work to time spent performing I/O, write bandwidth to the underlying storage system must increase proportionally to this increase in the checkpoint and computation size. Unfortunately, popular scientific self-describing file formats such as netCDF and HDF5 are designed with a focus on portability and flexibility. Extra care and careful craftingmore »« less
Design and Implementation of Burst Buffer Over-Subscription Scheme for HPC Storage Systems

Journal Article Bang, Jiwoo ; Sim, Alexander ; Lockwood, Glenn K. ; ... - IEEE Access

Burst Buffer is widely used in supercomputer centers to bridge the performance gap between computational power and the high-performance I/O systems. The primary role of Burst Buffer is to temporarily absorb the bursty I/O and reduce the heavy access on Parallel File System (PFS). However, the job resource manager on High-Performance Computer (HPC) systems prefers to use a dedicated Burst Buffer allocation approach, which eventually leads to the severely underutilized Burst Buffer resource. To improve the efficiency of using the expensive Burst Buffer resource, we analyze the I/O patterns on Burst Buffer in depth. We propose Burst Buffer over-subscription allocationmore »« less
https://doi.org/10.1109/ACCESS.2022.3233829

Similar Records