Parallel compression of data chunks of a shared data object using a log-structured file system

Bent, John M.; Faibish, Sorin; Grider, Gary

Advanced Search OptionsAdvanced Search queries use a traditional Term Search. For more info, see our FAQ.

All Fields:

Patent Title:

Abstract:

Assignee:

Inventor(s):

Patent Number:

Patent Classification (CPC):

All Classifications
A - human necessities
A01 - agriculture
A21 - baking
A22 - butchering
A23 - foods or foodstuffs
A24 - tobacco
A41 - wearing apparel
A42 - headwear
A43 - footwear
A44 - haberdashery
A45 - hand or travelling articles
A46 - brushware
A47 - furniture
A61 - medical or veterinary science
A62 - life-saving
A63 - sports
A99 - subject matter not otherwise provided for in this section
B - performing operations
B01 - physical or chemical processes or apparatus in general
B02 - crushing, pulverising, or disintegrating
B03 - separation of solid materials using liquids or using pneumatic tables or jigs
B04 - centrifugal apparatus or machines for carrying-out physical or chemical processes
B05 - spraying or atomising in general
B06 - generating or transmitting mechanical vibrations in general
B07 - separating solids from solids
B08 - cleaning
B09 - disposal of solid waste
B21 - mechanical metal-working without essentially removing material
B22 - casting
B23 - machine tools
B24 - grinding
B25 - hand tools
B26 - hand cutting tools
B27 - working or preserving wood or similar material
B28 - working cement, clay, or stone
B29 - working of plastics
B30 - presses
B31 - making articles of paper, cardboard or material worked in a manner analogous to paper
B32 - layered products
B33 - additive manufacturing technology
B41 - printing
B42 - bookbinding
B43 - writing or drawing implements
B44 - decorative arts
B60 - vehicles in general
B61 - railways
B62 - land vehicles for travelling otherwise than on rails
B63 - ships or other waterborne vessels
B64 - aircraft
B65 - conveying
B66 - hoisting
B67 - opening, closing {or cleaning} bottles, jars or similar containers
B68 - saddlery
B81 - microstructural technology
B82 - nanotechnology
B99 - subject matter not otherwise provided for in this section
C - chemistry
C01 - inorganic chemistry
C02 - treatment of water, waste water, sewage, or sludge
C03 - glass
C04 - cements
C05 - fertilisers
C06 - explosives
C07 - organic chemistry
C08 - organic macromolecular compounds
C09 - dyes
C10 - petroleum, gas or coke industries
C11 - animal or vegetable oils, fats, fatty substances or waxes
C12 - biochemistry
C13 - sugar industry
C14 - skins
C21 - metallurgy of iron
C22 - metallurgy
C23 - coating metallic material
C25 - electrolytic or electrophoretic processes
C30 - crystal growth
C40 - combinatorial technology
C99 - subject matter not otherwise provided for in this section
D - textiles
D01 - natural or man-made threads or fibres
D02 - yarns
D03 - weaving
D04 - braiding
D05 - sewing
D06 - treatment of textiles or the like
D07 - ropes
D10 - indexing scheme associated with sublasses of section d, relating to textiles
D21 - paper-making
D99 - subject matter not otherwise provided for in this section
E - fixed constructions
E01 - construction of roads, railways, or bridges
E02 - hydraulic engineering
E03 - water supply
E04 - building
E05 - locks
E06 - doors, windows, shutters, or roller blinds in general
E21 - earth drilling
E99 - subject matter not otherwise provided for in this section
F - mechanical engineering
F01 - machines or engines in general
F02 - combustion engines
F03 - machines or engines for liquids
F04 - positive - displacement machines for liquids
F05 - indexing schemes relating to engines or pumps in various subclasses of classes f01-f04
F15 - fluid-pressure actuators
F16 - engineering elements and units
F17 - storing or distributing gases or liquids
F21 - lighting
F22 - steam generation
F23 - combustion apparatus
F24 - heating
F25 - refrigeration or cooling
F26 - drying
F27 - furnaces
F28 - heat exchange in general
F41 - weapons
F42 - ammunition
F99 - subject matter not otherwise provided for in this section
G - physics
G01 - measuring
G02 - optics
G03 - photography
G04 - horology
G05 - controlling
G06 - computing
G07 - checking-devices
G08 - signalling
G09 - education
G10 - musical instruments
G11 - information storage
G12 - instrument details
G16 - information and communication technology [ict] specially adapted for specific application fields
G21 - nuclear physics
G99 - subject matter not otherwise provided for in this section
H - electricity
H01 - basic electric elements
H02 - generation
H03 - basic electronic circuitry
H04 - electric communication technique
H05 - electric techniques not otherwise provided for
H99 - subject matter not otherwise provided for in this section
Y - new / cross sectional technologies
Y02 - technologies or applications for mitigation or adaptation against climate change
Y04 - information or communication technologies having an impact on other technology areas
Y10 - technical subjects covered by former uspc

More Options ...

Title: Parallel compression of data chunks of a shared data object using a log-structured file system

Abstract

Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.

Inventors:: Bent, John M.; Faibish, Sorin; Grider, Gary

Issue Date:: Tue Oct 25 00:00:00 EDT 2016

Research Org.:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1330322

Patent Number(s):: 9477682

Application Number:: 13/799,228

Assignee:: EMC Corporation (Hopkinton, MA); Los Alamos National Security, LLC (Los Alamos, NM)

Patent Classifications (CPCs):: G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING

H - ELECTRICITY H04 - ELECTRIC COMMUNICATION TECHNIQUE H04L - TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION

Show more

G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
G06F16/1858 - {Parallel file systems, i.e. file systems supporting multiple processors}
G06F16/2365 - {Ensuring data consistency and integrity}
G06F3/067 - {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}

H - ELECTRICITY H04 - ELECTRIC COMMUNICATION TECHNIQUE H04L - TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
H04L67/10 - {in which an application is distributed across nodes in the network
H04L67/1097 - {for distributed storage of data in a network, e.g. network file system [NFS], transport mechanisms for storage area networks [SAN] or network attached storage [NAS]
H04L69/04 - {Protocols for data compression

Show less

DOE Contract Number:: AC52-06NA25396

Resource Type:: Patent

Resource Relation:: Patent File Date: 2013 Mar 13

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Bent, John M., Faibish, Sorin, and Grider, Gary. Parallel compression of data chunks of a shared data object using a log-structured file system.  United States: N. p., 2016. 
        Web.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, & Grider, Gary. Parallel compression of data chunks of a shared data object using a log-structured file system.  United States.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, and Grider, Gary. Tue .  
        "Parallel compression of data chunks of a shared data object using a log-structured file system".  United States.  https://www.osti.gov/servlets/purl/1330322.

Copy to clipboard


                    
@article{osti_1330322,

  title        = {Parallel compression of data chunks of a shared data object using a log-structured file system},

  author       = {Bent, John M. and Faibish, Sorin and Grider, Gary},

  abstractNote = {Techniques are provided for parallel compression of data chunks being written to a shared object. A client executing on a compute node or a burst buffer node in a parallel computing system stores a data chunk generated by the parallel computing system to a shared data object on a storage node by compressing the data chunk; and providing the data compressed data chunk to the storage node that stores the shared object. The client and storage node may employ Log-Structured File techniques. The compressed data chunk can be de-compressed by the client when the data chunk is read. A storage node stores a data chunk as part of a shared object by receiving a compressed version of the data chunk from a compute node; and storing the compressed version of the data chunk to the shared data object on the storage node.},

  doi          = {},

  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Tue Oct 25 00:00:00 EDT 2016},

  month        = {Tue Oct 25 00:00:00 EDT 2016}

}

Copy to clipboard

Patent:

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Active Non-Volatile Memory Post-Processing
patent-application, August 2013

Kannan, Sudarsun; Milojicic, Dejan S.; Talwar, Vanish
US Patent Application 13/404619; 20130227194
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20130227194

PLFS: a checkpoint filesystem for parallel applications
conference, January 2009

Bent, John; Gibson, Garth; Grider, Gary
https://doi.org/10.1145/1654059.1654081

Using Active NVRAM for Cloud I/O
conference, October 2011

Kannan, Sudarsun; Milojicic, Dejan; Talwar, Vanish
2011 6th Open Cirrus Summit (OCS), 2011 Sixth Open Cirrus Summit
https://doi.org/10.1109/OCS.2011.12

An Evolutionary Path to Object Storage Access
conference, November 2012

Goodell, David; Kim, Seong Jo; Latham, Robert
2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
https://doi.org/10.1109/SC.Companion.2012.17

Similar Records in DOE Patents and OSTI.GOV collections:

Parallel checksumming of data chunks of a shared data object using a log-structured file system

Patent Bent, John M.; Faibish, Sorin; Grider, Gary

Checksum values are generated and used to verify the data integrity. A client executing in a parallel computing system stores a data chunk to a shared data object on a storage node in the parallel computing system. The client determines a checksum value for the data chunk; and provides the checksum value with the data chunk to the storage node that stores the shared object. The data chunk can be stored on the storage node with the corresponding checksum value as part of the shared object. The storage node may be part of a Parallel Log-Structured File System (PLFS), andmore » « less
Full Text Available
Storage of sparse files using parallel log-structured file system

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

A sparse file is stored without holes by storing a data portion of the sparse file using a parallel log-structured file system; and generating an index entry for the data portion, the index entry comprising a logical offset, physical offset and length of the data portion. The holes can be restored to the sparse file upon a reading of the sparse file. The data portion can be stored at a logical end of the sparse file. Additional storage efficiency can optionally be achieved by (i) detecting a write pattern for a plurality of the data portions and generating a singlemore » « less
Full Text Available
Pattern-aware prefetching using parallel log-structured file system

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Techniques are provided for pattern-aware prefetching using a parallel log-structured file system. At least a portion of one or more files is accessed by detecting at least one pattern in a non-sequential access of the one or more files; and obtaining at least a portion of the one or more files based on the detected at least one pattern. The obtaining step comprises, for example, a prefetching or pre-allocation of the at least the portion of the one or more files. A prefetch cache can store the portion of the one or more obtained files. The cached portion of themore » « less
Full Text Available
Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data

Patent Grider, Gary A.; Poole, Stephen W.

Collective buffering and data pattern solutions are provided for storage, retrieval, and/or analysis of data in a collective parallel processing environment. For example, a method can be provided for data storage in a collective parallel processing environment. The method comprises receiving data to be written for a plurality of collective processes within a collective parallel processing environment, extracting a data pattern for the data to be written for the plurality of collective processes, generating a representation describing the data pattern, and saving the data and the representation.
Full Text Available
Global distributed file append using log-structured file system

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Techniques are provided for globally appending data from a group of distributed processes to a shared file using a log-structured file system. Data generated by a plurality of processes in a parallel computing system are appended to a shared file by storing the data to the shared file using a log-structured file system (such as a Parallel Log-Structured File System (PLFS)); and generating an index entry for the data, the index entry comprising a logical offset entry and a timestamp entry indicating a time of the storage, wherein the logical offset entry is resolved at read time. The logical offsetmore » « less
Full Text Available

Similar Records

Title: Parallel compression of data chunks of a shared data object using a log-structured file system

Abstract

Citation Formats

Active Non-Volatile Memory Post-Processing patent-application, August 2013

PLFS: a checkpoint filesystem for parallel applications conference, January 2009

Using Active NVRAM for Cloud I/O conference, October 2011

An Evolutionary Path to Object Storage Access conference, November 2012

Active Non-Volatile Memory Post-Processing
patent-application, August 2013

PLFS: a checkpoint filesystem for parallel applications
conference, January 2009

Using Active NVRAM for Cloud I/O
conference, October 2011

An Evolutionary Path to Object Storage Access
conference, November 2012