skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Exploiting Lustre File Joining for Effective Collective IO

Abstract

Lustre is a parallel file system that presents high aggregated IO bandwidth by striping file extents across many storage devices. However, our experiments indicate excessively wide striping can cause performance degradation. Lustre supports an innovative file joining feature that joins files in place. To mitigate striping overhead and benefit collective IO, we propose two techniques: split writing and hierarchical striping. In split writing, a file is created as separate subfiles, each of which is striped to only a few storage devices. They are joined as a single file at the file close time. Hierarchical striping builds on top of split writing and orchestrates the span of subfiles in a hierarchical manner to avoid overlapping and achieve the appropriate coverage of storage devices. Together, these techniques can avoid the overhead associated with large stripe width, while still being able to combine bandwidth available from many storage devices. We have prototyped these techniques in the ROMIO implementation of MPI-IO. Experimental results indicate that split writing and hierarchical striping can significantly improve the performance of Lustre collective IO in terms of both data transfer and management operations. On a Lustre file system configured with 46 object storage targets, our implementation improves collective writemore » performance of a 16-process job by as much as 220%.« less

Authors:
 [1];  [1];  [1];  [2]
  1. ORNL
  2. Wayne State University, Detroit
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Center for Computational Sciences
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
931507
DOE Contract Number:  
DE-AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: Seventh IEEE International Symposium on Cluster Computing and the Grid CCGrid 2007, Rio de Janeiro - Brazil, Brazil, 20070514, 20070517
Country of Publication:
United States
Language:
English
Subject:
99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; IMPLEMENTATION; MANAGEMENT; PERFORMANCE; STORAGE; TARGETS; COMPUTERS

Citation Formats

Yu, Weikuan, Vetter, Jeffrey S, Canon, Richard Shane, and Jiang, Song. Exploiting Lustre File Joining for Effective Collective IO. United States: N. p., 2007. Web.
Yu, Weikuan, Vetter, Jeffrey S, Canon, Richard Shane, & Jiang, Song. Exploiting Lustre File Joining for Effective Collective IO. United States.
Yu, Weikuan, Vetter, Jeffrey S, Canon, Richard Shane, and Jiang, Song. Mon . "Exploiting Lustre File Joining for Effective Collective IO". United States. doi:.
@article{osti_931507,
title = {Exploiting Lustre File Joining for Effective Collective IO},
author = {Yu, Weikuan and Vetter, Jeffrey S and Canon, Richard Shane and Jiang, Song},
abstractNote = {Lustre is a parallel file system that presents high aggregated IO bandwidth by striping file extents across many storage devices. However, our experiments indicate excessively wide striping can cause performance degradation. Lustre supports an innovative file joining feature that joins files in place. To mitigate striping overhead and benefit collective IO, we propose two techniques: split writing and hierarchical striping. In split writing, a file is created as separate subfiles, each of which is striped to only a few storage devices. They are joined as a single file at the file close time. Hierarchical striping builds on top of split writing and orchestrates the span of subfiles in a hierarchical manner to avoid overlapping and achieve the appropriate coverage of storage devices. Together, these techniques can avoid the overhead associated with large stripe width, while still being able to combine bandwidth available from many storage devices. We have prototyped these techniques in the ROMIO implementation of MPI-IO. Experimental results indicate that split writing and hierarchical striping can significantly improve the performance of Lustre collective IO in terms of both data transfer and management operations. On a Lustre file system configured with 46 object storage targets, our implementation improves collective write performance of a 16-process job by as much as 220%.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Mon Jan 01 00:00:00 EST 2007},
month = {Mon Jan 01 00:00:00 EST 2007}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: