Parallel file system with metadata distributed across partitioned key-value store c
Abstract
Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1393429
- Patent Number(s):
- 9767107
- Application Number:
- 13/931,832
- Assignee:
- EMC IP Holding Company LLC
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Resource Relation:
- Patent File Date: 2013 Jun 29
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c. United States: N. p., 2017.
Web.
Bent, John M., Faibish, Sorin, Grider, Gary, & Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c. United States.
Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Tue .
"Parallel file system with metadata distributed across partitioned key-value store c". United States. https://www.osti.gov/servlets/purl/1393429.
@article{osti_1393429,
title = {Parallel file system with metadata distributed across partitioned key-value store c},
author = {Bent, John M. and Faibish, Sorin and Grider, Gary and Torres, Aaron},
abstractNote = {Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2017},
month = {9}
}
Works referenced in this record:
Parallel File System Analysis Through Application I/O Tracing
journal, May 2012
- Wright, S. A.; Hammond, S. D.; Pennycook, S. J.
- The Computer Journal, Vol. 56, Issue 2, p. 141-155