skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves

Abstract

Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many stages of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in greater than expected end-to-end I/O latency. In this paper, we present the design and implementation of a novel data management infrastructure called Software-Defined Storage Resource Enclaves (SIREN) at system level to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Lastly, our experimental results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers (burst buffer and parallel file systems) while maintaining high system scalability and resource utilization.

Authors:
 [1];  [1];  [1]; ORCiD logo [2];  [1]
  1. Washington State Univ., Vancouver, WA (United States)
  2. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1482940
Report Number(s):
LA-UR-18-22116
Journal ID: ISSN 2372-207X
Grant/Contract Number:  
89233218CNA000001
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
IEEE Transactions on Multi-Scale Computing Systems
Additional Journal Information:
Journal Name: IEEE Transactions on Multi-Scale Computing Systems; Journal ID: ISSN 2372-207X
Publisher:
IEEE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Computer Science; QoS; Software-Defined Storage; Scientific Workflows

Citation Formats

Karki, Suman, Nguyen, Bao, Feener, Joshua, Davis, Kei Marion, and Zhang, Xuechen. Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves. United States: N. p., 2018. Web. doi:10.1109/TMSCS.2018.2879096.
Karki, Suman, Nguyen, Bao, Feener, Joshua, Davis, Kei Marion, & Zhang, Xuechen. Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves. United States. doi:10.1109/TMSCS.2018.2879096.
Karki, Suman, Nguyen, Bao, Feener, Joshua, Davis, Kei Marion, and Zhang, Xuechen. Thu . "Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves". United States. doi:10.1109/TMSCS.2018.2879096.
@article{osti_1482940,
title = {Enforcing End-to-end I/O Policies for Scientific Workflows using Software-Defined Storage Resource Enclaves},
author = {Karki, Suman and Nguyen, Bao and Feener, Joshua and Davis, Kei Marion and Zhang, Xuechen},
abstractNote = {Data-intensive knowledge discovery requires scientific applications to run concurrently with analytics and visualization codes executing in situ for timely output inspection and knowledge extraction. Consequently, I/O pipelines of scientific workflows can be long and complex because they comprise many stages of analytics across different layers of the I/O stack of high-performance computing systems. Performance limitations at any I/O layer or stage can cause an I/O bottleneck resulting in greater than expected end-to-end I/O latency. In this paper, we present the design and implementation of a novel data management infrastructure called Software-Defined Storage Resource Enclaves (SIREN) at system level to enforce end-to-end policies that dictate an I/O pipeline's performance. SIREN provides an I/O performance interface for users to specify the desired storage resources in the context of in-situ analytics. If suboptimal performance of analytics is caused by an I/O bottleneck when data are transferred between simulations and analytics, schedulers in different layers of the I/O stack automatically provide the guaranteed lower bounds on I/O throughput. Lastly, our experimental results demonstrate that SIREN provides performance isolation among scientific workflows sharing multiple storage servers across two I/O layers (burst buffer and parallel file systems) while maintaining high system scalability and resource utilization.},
doi = {10.1109/TMSCS.2018.2879096},
journal = {IEEE Transactions on Multi-Scale Computing Systems},
issn = {2372-207X},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on November 1, 2019
Publisher's Version of Record

Save / Share: