Small file aggregation in a parallel computing system
Abstract
Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1151820
- Patent Number(s):
- 8825652
- Application Number:
- 13/536,315
- Assignee:
- EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Small file aggregation in a parallel computing system. United States: N. p., 2014.
Web.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, & Zhang, Jingwang. Small file aggregation in a parallel computing system. United States.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Tue .
"Small file aggregation in a parallel computing system". United States. https://www.osti.gov/servlets/purl/1151820.
@article{osti_1151820,
title = {Small file aggregation in a parallel computing system},
author = {Faibish, Sorin and Bent, John M. and Tzelnic, Percy and Grider, Gary and Zhang, Jingwang},
abstractNote = {Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Sep 02 00:00:00 EDT 2014},
month = {Tue Sep 02 00:00:00 EDT 2014}
}
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Document storing and managing system
patent, October 1998
- Nakagawa, Aki; Kanno, Yuji; Hata, Tsutomu
- US Patent Document 5,819,295
Write anywhere file-system layout
patent, October 1999
- Hitz, David; Malcolm, Michael; Lau, James
- US Patent Document 5,963,962
Storage management system with file aggregation
patent, August 2000
- Cannon, David M.; Kaczmarski, Michael Allen
- US Patent Document 6,098,074
Rule based aggregation of files and transactions in a switched file system
patent-application, July 2004
- Miloushev, Vladimir; Nickolov, Peter
- US Patent Application 10/336832; 20040133577
Maintaining an aggregate including active files in a storage pool
patent-application, February 2007
- Cannon, David Maxwell; Martin, Howard Newton
- US Patent Application 11/206488; 20070043785
Maintaining active-only copy storage pools
patent-application, February 2007
- Cannon, David Maxwell; Martin, Howard Newton
- US Patent Application 11/224852; 20070043789
PLFS: a checkpoint filesystem for parallel applications
conference, January 2009
- Bent, John; Gibson, Garth; Grider, Gary
Works referencing / citing this record:
Storing data files in a file system
patent, May 2016
- Akelbein, Jens-Peter; Haustein, Nils
- US Patent Document 9,355,108
Architecture and method for a burst buffer using flash technology
patent, March 2016
- Tzelnic, Percy; Faibish, Sorin; Gupta, Uday
- US Patent Document 9,286,261
Burst buffer appliance with small file aggregation
patent, March 2015
- Faibish, Sorin; Bent, John M.
- US Patent Document 8,972,465