Computing erasure metadata and data layout prior to storage using a processing platform

Bent, John M.; Faibish, Sorin; Ting, Dennis P.J.; Teymouri, Sasan; Pedone, James M.; Grider, Gary

Title: Computing erasure metadata and data layout prior to storage using a processing platform

Abstract

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

Inventors:: Bent, John M.; Faibish, Sorin; Ting, Dennis P.J.; Teymouri, Sasan; Pedone, Jr., James M.; Grider, Gary

Issue Date:: 2018-11-20

Research Org.:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1494131

Patent Number(s):: 10135924

Application Number:: 14/751,372

Assignee:: EMC IP Holding Company LLC (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)

Patent Classifications (CPCs):: G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING

H - ELECTRICITY H04 - ELECTRIC COMMUNICATION TECHNIQUE H04L - TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION

Show more

G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
G06F3/067 - {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}

H - ELECTRICITY H04 - ELECTRIC COMMUNICATION TECHNIQUE H04L - TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
H04L67/1097 - {for distributed storage of data in a network, e.g. network file system [NFS], transport mechanisms for storage area networks [SAN] or network attached storage [NAS]

G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
G06F15/167 - using a common memory, e.g. mailbox
G06F3/0689 - {Disk arrays, e.g. RAID, JBOD}
G06F3/064 - {Management of blocks}
G06F3/061 - {Improving I/O performance}
G06F3/0611 - {in relation to response time}

Show less

DOE Contract Number:: AC52-06NA25396

Resource Type:: Patent

Resource Relation:: Patent File Date: 2015 Jun 26

Country of Publication:: United States

Language:: English

Subject:: 96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Citation Formats


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform.  United States: N. p., 2018. 
        Web.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., & Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform.  United States.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Tue .  
        "Computing erasure metadata and data layout prior to storage using a processing platform".  United States.  https://www.osti.gov/servlets/purl/1494131.

Copy to clipboard


                    
@article{osti_1494131,

  title        = {Computing erasure metadata and data layout prior to storage using a processing platform},

  author       = {Bent, John M. and Faibish, Sorin and Ting, Dennis P.J. and Teymouri, Sasan and Pedone, Jr., James M. and Grider, Gary},

  abstractNote = {Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.},

  doi          = {},

  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {11}

}

Copy to clipboard

Patent:

View Patent

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Jitter-free co-processing on a prototype exascale storage stack
conference, April 2012

Bent, John; Faibish, Sorin; Ahrens, Jim
2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST)
https://doi.org/10.1109/MSST.2012.6232382

Similar Records in DOE Patents and OSTI.GOV collections:

Method and system for semantic metadata compression in a two-tier storage system using copy-on-write

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Example embodiments of the present invention relate to a method, a system, and a computer program product for managing metadata. The method includes determining a common characteristic of metadata entries for a plurality of files in a directory of a file system and computing a representative metadata entry for the plurality of metadata entries according to the common characteristic of the metadata entries. The plurality of metadata entries in the file system then may be represented according to the computed representative metadata entry.
Full Text Available
Cloud object store for archive storage of high performance computing data using decoupling middleware

Patent Bent, John M.; Faibish, Sorin; Grider, Gary

Cloud object storage is enabled for archived data, such as checkpoints and results, of high performance computing applications using a middleware process. A plurality of archived files, such as checkpoint files and results, generated by a plurality of processes in a parallel computing system are stored by obtaining the plurality of archived files from the parallel computing system; converting the plurality of archived files to objects using a log structured file system middleware process; and providing the objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes.more » « less
Full Text Available
Method and system for reducing metadata overhead in a two-tier storage architecture

Patent Bent, John M.; Faibish, Sorin; Ting, Dennis Pei Jean; ...

Example embodiments of the present invention relate to a method, a system, and a computer program product for maintaining dynamically loadable namespaces. The method includes presenting, to an application, a file system having a global single namespace (SN) comprising a plurality of dynamically-loadable namespaces (DLNs) maintained in a capacity tier (CT) of object storage of a storage system. In response to a job submitted by the application, dynamically loading a first dynamically-loadable namespace, including the metadata only for the first dynamically-loadable namespace, associated with the job from the capacity tier of object storage into a fast tier (FT) of storagemore » of the storage system. « less
Full Text Available
Distributed metadata servers for cluster file systems using shared low latency persistent key-value metadata store

Patent Bent, John M.; Faibish, Sorin; Pedone, Jr., James M.; ...

A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata requestmore » « less
Full Text Available
Distributed metadata in a high performance computing environment

Patent Bent, John M.; Faibish, Sorin; Zhang, Zhenhua; ...

A computer-executable method, system, and computer program product for managing meta-data in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store, the co computer-executable method, system, and computer program product comprising receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value, determining which of the one or more burst buffers stores the requested metadata, and upon determination thatmore » « less
Full Text Available

Similar Records

Title: Computing erasure metadata and data layout prior to storage using a processing platform

Abstract

Citation Formats

Jitter-free co-processing on a prototype exascale storage stack conference, April 2012

Jitter-free co-processing on a prototype exascale storage stack
conference, April 2012