DOE Patents title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Small file aggregation in a parallel computing system

Abstract

Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.

Inventors:
; ; ; ;
Issue Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1151820
Patent Number(s):
8825652
Application Number:
13/536,315
Assignee:
EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
Patent Classifications (CPCs):
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
DOE Contract Number:  
AC52-06NA25396
Resource Type:
Patent
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Small file aggregation in a parallel computing system. United States: N. p., 2014. Web.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, & Zhang, Jingwang. Small file aggregation in a parallel computing system. United States.
Faibish, Sorin, Bent, John M., Tzelnic, Percy, Grider, Gary, and Zhang, Jingwang. Tue . "Small file aggregation in a parallel computing system". United States. https://www.osti.gov/servlets/purl/1151820.
@article{osti_1151820,
title = {Small file aggregation in a parallel computing system},
author = {Faibish, Sorin and Bent, John M. and Tzelnic, Percy and Grider, Gary and Zhang, Jingwang},
abstractNote = {Techniques are provided for small file aggregation in a parallel computing system. An exemplary method for storing a plurality of files generated by a plurality of processes in a parallel computing system comprises aggregating the plurality of files into a single aggregated file; and generating metadata for the single aggregated file. The metadata comprises an offset and a length of each of the plurality of files in the single aggregated file. The metadata can be used to unpack one or more of the files from the single aggregated file.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2014},
month = {9}
}

Works referenced in this record:

Document storing and managing system
patent, October 1998


Write anywhere file-system layout
patent, October 1999


Storage management system with file aggregation
patent, August 2000


Rule based aggregation of files and transactions in a switched file system
patent-application, July 2004


Maintaining an aggregate including active files in a storage pool
patent-application, February 2007


Maintaining active-only copy storage pools
patent-application, February 2007


PLFS: a checkpoint filesystem for parallel applications
conference, January 2009


    Works referencing / citing this record:

    Storing data files in a file system
    patent, May 2016


    Architecture and method for a burst buffer using flash technology
    patent, March 2016


    Burst buffer appliance with small file aggregation
    patent, March 2015