Global distributed file append using log-structured file system
Abstract
Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1568208
- Patent Number(s):
- 10262000
- Application Number:
- 13/921,657
- Assignee:
- EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Resource Relation:
- Patent File Date: 06/19/2013
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 99 GENERAL AND MISCELLANEOUS
Citation Formats
Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Global distributed file append using log-structured file system. United States: N. p., 2019.
Web.
Bent, John M., Faibish, Sorin, Grider, Gary, & Torres, Aaron. Global distributed file append using log-structured file system. United States.
Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Tue .
"Global distributed file append using log-structured file system". United States. https://www.osti.gov/servlets/purl/1568208.
@article{osti_1568208,
title = {Global distributed file append using log-structured file system},
author = {Bent, John M. and Faibish, Sorin and Grider, Gary and Torres, Aaron},
abstractNote = {Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2019},
month = {4}
}
Works referenced in this record:
Small file aggregation in a parallel computing system
patent, September 2014
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 8,825,652