skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Parallel file system with metadata distributed across partitioned key-value store c

Abstract

Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).

Inventors:
; ; ;
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1393429
Patent Number(s):
9,767,107
Application Number:
13/931,832
Assignee:
EMC IP Holding Company LLC LANL
DOE Contract Number:
AC52-06NA25396
Resource Type:
Patent
Resource Relation:
Patent File Date: 2013 Jun 29
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c. United States: N. p., 2017. Web.
Bent, John M., Faibish, Sorin, Grider, Gary, & Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c. United States.
Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Tue . "Parallel file system with metadata distributed across partitioned key-value store c". United States. doi:. https://www.osti.gov/servlets/purl/1393429.
@article{osti_1393429,
title = {Parallel file system with metadata distributed across partitioned key-value store c},
author = {Bent, John M. and Faibish, Sorin and Grider, Gary and Torres, Aaron},
abstractNote = {Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Sep 19 00:00:00 EDT 2017},
month = {Tue Sep 19 00:00:00 EDT 2017}
}

Patent:

Save / Share: