skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Parallel compression of data chunks of a shared data object using a log-structured file system

Abstract

Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.

Inventors:
; ;
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1330322
Patent Number(s):
9,477,682
Application Number:
13/799,228
Assignee:
EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM) LANL
DOE Contract Number:  
AC52-06NA25396
Resource Type:
Patent
Resource Relation:
Patent File Date: 2013 Mar 13
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Bent, John M., Faibish, Sorin, and Grider, Gary. Parallel compression of data chunks of a shared data object using a log-structured file system. United States: N. p., 2016. Web.
Bent, John M., Faibish, Sorin, & Grider, Gary. Parallel compression of data chunks of a shared data object using a log-structured file system. United States.
Bent, John M., Faibish, Sorin, and Grider, Gary. Tue . "Parallel compression of data chunks of a shared data object using a log-structured file system". United States. doi:. https://www.osti.gov/servlets/purl/1330322.
@article{osti_1330322,
title = {Parallel compression of data chunks of a shared data object using a log-structured file system},
author = {Bent, John M. and Faibish, Sorin and Grider, Gary},
abstractNote = {Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Oct 25 00:00:00 EDT 2016},
month = {Tue Oct 25 00:00:00 EDT 2016}
}

Patent:

Save / Share:

Works referenced in this record:

PLFS: a checkpoint filesystem for parallel applications
conference, January 2009