skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Efficient structured data access in parallel file systems.

Abstract

Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel ?le systems do not traditionally support such access. In this work we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support 'datatype I/O' because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation.

Authors:
; ; ; ;
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC); National Aeronautic and Space Administration (NASA)
OSTI Identifier:
925108
Report Number(s):
ANL/MCS/CP-111484
TRN: US200807%%44
DOE Contract Number:  
DE-AC02-06CH11357
Resource Type:
Conference
Resource Relation:
Conference: Cluster 2003: IEEE International Conference on Cluster Computing; Dec 1-4, 2003; Hong Kong
Country of Publication:
United States
Language:
ENGLISH
Subject:
99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; PARALLEL PROCESSING; DATA BASE MANAGEMENT; INFORMATION RETRIEVAL; EFFICIENCY

Citation Formats

Ching, A, Choudhary, A, Liao, W -K, Ross, R, Gropp, W, Mathematics and Computer Science, and Northwestern Univ. Efficient structured data access in parallel file systems.. United States: N. p., 2003. Web.
Ching, A, Choudhary, A, Liao, W -K, Ross, R, Gropp, W, Mathematics and Computer Science, & Northwestern Univ. Efficient structured data access in parallel file systems.. United States.
Ching, A, Choudhary, A, Liao, W -K, Ross, R, Gropp, W, Mathematics and Computer Science, and Northwestern Univ. Wed . "Efficient structured data access in parallel file systems.". United States.
@article{osti_925108,
title = {Efficient structured data access in parallel file systems.},
author = {Ching, A and Choudhary, A and Liao, W -K and Ross, R and Gropp, W and Mathematics and Computer Science and Northwestern Univ.},
abstractNote = {Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel ?le systems do not traditionally support such access. In this work we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support 'datatype I/O' because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2003},
month = {1}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: