Computing erasure metadata and data layout prior to storage using a processing platform
Techniques are provided for computing data and metadata layout prior to storage in a storage system using a processing platform. An exemplary processing platform comprises one or more of a compute node and a burst buffer appliance. The processing platform communicates with a plurality of the compute nodes over a network, wherein a plurality of applications executing on the plurality of compute nodes generate a plurality of data objects; computes erasure metadata for one or more of the data objects on at least one of the compute nodes; and provides the erasure metadata with the corresponding one or more data objects to a storage system. The processing platform optionally determines a full set of the data objects to be stored and queries the storage system to determine an anticipated layout of the full set of the data objects to be stored. The anticipated layout allows special handling, for example, for small files and large files that are identified based on predefined criteria.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- Assignee:
- EMC IP Holding Company LLC (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)
- Patent Number(s):
- 10,135,924
- Application Number:
- 14/751,372
- OSTI ID:
- 1494131
- Resource Relation:
- Patent File Date: 2015 Jun 26
- Country of Publication:
- United States
- Language:
- English
Burst buffer appliance with small file aggregation
|
patent | March 2015 |
Jitter-free co-processing on a prototype exascale storage stack
|
conference | April 2012 |
Similar Records
Parallel file system with metadata distributed across partitioned key-value store c
MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing