Global distributed file append using log-structured file system
Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offset entry can be populated with an append placeholder that is resolved when the shared file is read. At read time, a plurality of the index entries associated with the shared file can be sorted using the timestamp entry to deliver the requested shared file to a requesting application.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- Assignee:
- EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
- Patent Number(s):
- 10,262,000
- Application Number:
- 13/921,657
- OSTI ID:
- 1568208
- Resource Relation:
- Patent File Date: 06/19/2013
- Country of Publication:
- United States
- Language:
- English
Small file aggregation in a parallel computing system
|
patent | September 2014 |
Similar Records
...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats
Parallel checksumming of data chunks of a shared data object using a log-structured file system