Small file aggregation in a parallel computing system
Abstract
Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.
- Inventors:
- Issue Date:
- Research Org.:
- LANL (Los Alamos National Laboratory (LANL), Los Alamos, NM (United States))
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1151820
- Patent Number(s):
- 8,825,652
- Application Number:
- 13/536,315
- Assignee:
- EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM) LANL
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Small file aggregation in a parallel computing system. United States: N. p., 2014.
Web.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, & Zhang, Jingwang. Small file aggregation in a parallel computing system. United States.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Tue .
"Small file aggregation in a parallel computing system". United States. https://www.osti.gov/servlets/purl/1151820.
@article{osti_1151820,
title = {Small file aggregation in a parallel computing system},
author = {Faibish, Sorin and Bent, John M. and Tzelnic, Percy and Grider, Gary and Zhang, Jingwang},
abstractNote = {Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2014},
month = {9}
}
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
PLFS: a checkpoint filesystem for parallel applications
conference, January 2009
- Bent, John; Gibson, Garth; Grider, Gary