Computing erasure metadata and data layout prior to storage using a processing platform

Title: Computing erasure metadata and data layout prior to storage using a processing platform

Full Record
Other Related Research

Abstract

Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.

Inventors:: Bent, John M.; Faibish, Sorin; Ting, Dennis P.J.; Teymouri, Sasan; Pedone, Jr., James M.; Grider, Gary

Issue Date:: 2018-11-20

Research Org.:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1494131

Patent Number(s):: 10,135,924

Application Number:: 14/751,372

Assignee:: EMC IP Holding Company LLC (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM) LANL

DOE Contract Number:: AC52-06NA25396

Resource Type:: Patent

Resource Relation:: Patent File Date: 2015 Jun 26

Country of Publication:: United States

Language:: English

Subject:: 96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Citation Formats


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform.  United States: N. p., 2018. 
        Web.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., & Grider, Gary. Computing erasure metadata and data layout prior to storage using a processing platform.  United States.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Ting, Dennis P.J., Teymouri, Sasan, Pedone, Jr., James M., and Grider, Gary. Tue .  
        "Computing erasure metadata and data layout prior to storage using a processing platform".  United States.  https://www.osti.gov/servlets/purl/1494131.

Copy to clipboard


                    
@article{osti_1494131,

  title        = {Computing erasure metadata and data layout prior to storage using a processing platform},

  author       = {Bent, John M. and Faibish, Sorin and Ting, Dennis P.J. and Teymouri, Sasan and Pedone, Jr., James M. and Grider, Gary},

  abstractNote = {Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.},

  doi          = {},

  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {11}

}

Copy to clipboard

Patent:

View Patent

Save / Share:

Export Metadata

Save to My Library

Similar records in DOE Patents and OSTI.GOV collections:

Cloud object store for archive storage of high performance computing data using decoupling middleware

Patent Bent, John M. ; Faibish, Sorin ; Grider, Gary

Cloud object storage is enabled for archived data, such as checkpoints and results, of high performance computing applications using a middleware process. A plurality of archived files, such as checkpoint files and results, generated by a plurality of processes in a parallel computing system are stored by obtaining the plurality of archived files from the parallel computing system; converting the plurality of archived files to objects using a log structured file system middleware process; and providing the objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes.more »« less
Full Text Available
Distributed metadata servers for cluster file systems using shared low latency persistent key-value metadata store

Patent Bent, John M. ; Faibish, Sorin ; Pedone, Jr., James M. ; ...

A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata requestmore »« less
Full Text Available
Distributed metadata in a high performance computing environment

Patent Bent, John M. ; Faibish, Sorin ; Zhang, Zhenhua ; ...

A computer-executable method, system, and computer program product for managing meta-data in a distributed storage system, wherein the distributed storage system includes one or more burst buffers enabled to operate with a distributed key-value store, the co computer-executable method, system, and computer program product comprising receiving a request for meta-data associated with a block of data stored in a first burst buffer of the one or more burst buffers in the distributed storage system, wherein the meta data is associated with a key-value, determining which of the one or more burst buffers stores the requested metadata, and upon determination thatmore »« less
Full Text Available
Systems and methods for rapid processing and storage of data

Patent Stalzer, Mark A.

Systems and methods of building massively parallel computing systems using low power computing complexes in accordance with embodiments of the invention are disclosed. A massively parallel computing system in accordance with one embodiment of the invention includes at least one Solid State Blade configured to communicate via a high performance network fabric. In addition, each Solid State Blade includes a processor configured to communicate with a plurality of low power computing complexes interconnected by a router, and each low power computing complex includes at least one general processing core, an accelerator, an I/O interface, and cache memory and is configuredmore »« less
Full Text Available
Efficient accesses of data structures using processing near memory

Patent Jayasena, Nuwan S. ; Zhang, Dong Ping ; Diez, Paula Aguilera

Systems, apparatuses, and methods for implementing efficient queues and other data structures. A queue may be shared among multiple processors and/or threads without using explicit software atomic instructions to coordinate access to the queue. System software may allocate an atomic queue and corresponding queue metadata in system memory and return, to the requesting thread, a handle referencing the queue metadata. Any number of threads may utilize the handle for accessing the atomic queue. The logic for ensuring the atomicity of accesses to the atomic queue may reside in a management unit in the memory controller coupled to the memory wheremore » the atomic queue is allocated.« less
Full Text Available

Similar Records