Computing erasure metadata and data layout prior to storage using a processing platform
Abstract
Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1494131
- Patent Number(s):
- 10135924
- Application Number:
- 14/751,372
- Assignee:
- EMC IP Holding Company LLC (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
H - ELECTRICITY H04 - ELECTRIC COMMUNICATION TECHNIQUE H04L - TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Resource Relation:
- Patent File Date: 2015 Jun 26
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 96 KNOWLEDGE MANAGEMENT AND PRESERVATION
Citation Formats
Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform. United States: N. p., 2018.
Web.
Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., & Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform. United States.
Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Tue .
"Computing erasure metadata and data layout prior to storage using a processing platform". United States. https://www.osti.gov/servlets/purl/1494131.
@article{osti_1494131,
title = {Computing erasure metadata and data layout prior to storage using a processing platform},
author = {Bent, John M. and Faibish, Sorin and Ting, Dennis P.J. and Teymouri, Sasan and Pedone, Jr., James M. and Grider, Gary},
abstractNote = {Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {11}
}