Parallel file system with metadata distributed across partitioned key-value store c

Bent, John M.; Faibish, Sorin; Grider, Gary; Torres, Aaron

Advanced Search OptionsAdvanced Search queries use a traditional Term Search. For more info, see our FAQ.

All Fields:

Patent Title:

Abstract:

Assignee:

Inventor(s):

Patent Number:

Patent Classification (CPC):

All Classifications
A - human necessities
A01 - agriculture
A21 - baking
A22 - butchering
A23 - foods or foodstuffs
A24 - tobacco
A41 - wearing apparel
A42 - headwear
A43 - footwear
A44 - haberdashery
A45 - hand or travelling articles
A46 - brushware
A47 - furniture
A61 - medical or veterinary science
A62 - life-saving
A63 - sports
A99 - subject matter not otherwise provided for in this section
B - performing operations
B01 - physical or chemical processes or apparatus in general
B02 - crushing, pulverising, or disintegrating
B03 - separation of solid materials using liquids or using pneumatic tables or jigs
B04 - centrifugal apparatus or machines for carrying-out physical or chemical processes
B05 - spraying or atomising in general
B06 - generating or transmitting mechanical vibrations in general
B07 - separating solids from solids
B08 - cleaning
B09 - disposal of solid waste
B21 - mechanical metal-working without essentially removing material
B22 - casting
B23 - machine tools
B24 - grinding
B25 - hand tools
B26 - hand cutting tools
B27 - working or preserving wood or similar material
B28 - working cement, clay, or stone
B29 - working of plastics
B30 - presses
B31 - making articles of paper, cardboard or material worked in a manner analogous to paper
B32 - layered products
B33 - additive manufacturing technology
B41 - printing
B42 - bookbinding
B43 - writing or drawing implements
B44 - decorative arts
B60 - vehicles in general
B61 - railways
B62 - land vehicles for travelling otherwise than on rails
B63 - ships or other waterborne vessels
B64 - aircraft
B65 - conveying
B66 - hoisting
B67 - opening, closing {or cleaning} bottles, jars or similar containers
B68 - saddlery
B81 - microstructural technology
B82 - nanotechnology
B99 - subject matter not otherwise provided for in this section
C - chemistry
C01 - inorganic chemistry
C02 - treatment of water, waste water, sewage, or sludge
C03 - glass
C04 - cements
C05 - fertilisers
C06 - explosives
C07 - organic chemistry
C08 - organic macromolecular compounds
C09 - dyes
C10 - petroleum, gas or coke industries
C11 - animal or vegetable oils, fats, fatty substances or waxes
C12 - biochemistry
C13 - sugar industry
C14 - skins
C21 - metallurgy of iron
C22 - metallurgy
C23 - coating metallic material
C25 - electrolytic or electrophoretic processes
C30 - crystal growth
C40 - combinatorial technology
C99 - subject matter not otherwise provided for in this section
D - textiles
D01 - natural or man-made threads or fibres
D02 - yarns
D03 - weaving
D04 - braiding
D05 - sewing
D06 - treatment of textiles or the like
D07 - ropes
D10 - indexing scheme associated with sublasses of section d, relating to textiles
D21 - paper-making
D99 - subject matter not otherwise provided for in this section
E - fixed constructions
E01 - construction of roads, railways, or bridges
E02 - hydraulic engineering
E03 - water supply
E04 - building
E05 - locks
E06 - doors, windows, shutters, or roller blinds in general
E21 - earth drilling
E99 - subject matter not otherwise provided for in this section
F - mechanical engineering
F01 - machines or engines in general
F02 - combustion engines
F03 - machines or engines for liquids
F04 - positive - displacement machines for liquids
F05 - indexing schemes relating to engines or pumps in various subclasses of classes f01-f04
F15 - fluid-pressure actuators
F16 - engineering elements and units
F17 - storing or distributing gases or liquids
F21 - lighting
F22 - steam generation
F23 - combustion apparatus
F24 - heating
F25 - refrigeration or cooling
F26 - drying
F27 - furnaces
F28 - heat exchange in general
F41 - weapons
F42 - ammunition
F99 - subject matter not otherwise provided for in this section
G - physics
G01 - measuring
G02 - optics
G03 - photography
G04 - horology
G05 - controlling
G06 - computing
G07 - checking-devices
G08 - signalling
G09 - education
G10 - musical instruments
G11 - information storage
G12 - instrument details
G16 - information and communication technology [ict] specially adapted for specific application fields
G21 - nuclear physics
G99 - subject matter not otherwise provided for in this section
H - electricity
H01 - basic electric elements
H02 - generation
H03 - basic electronic circuitry
H04 - electric communication technique
H05 - electric techniques not otherwise provided for
H99 - subject matter not otherwise provided for in this section
Y - new / cross sectional technologies
Y02 - technologies or applications for mitigation or adaptation against climate change
Y04 - information or communication technologies having an impact on other technology areas
Y10 - technical subjects covered by former uspc

More Options ...

Title: Parallel file system with metadata distributed across partitioned key-value store c

Abstract

Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).

Inventors:: Bent, John M.; Faibish, Sorin; Grider, Gary; Torres, Aaron

Issue Date:: Tue Sep 19 00:00:00 EDT 2017

Research Org.:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1393429

Patent Number(s):: 9767107

Application Number:: 13/931,832

Assignee:: EMC IP Holding Company LLC

Patent Classifications (CPCs):: G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING

Show more

G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
G06F16/134 - {Distributed indices}
G06F16/1858 - {Parallel file systems, i.e. file systems supporting multiple processors}

Show less

DOE Contract Number:: AC52-06NA25396

Resource Type:: Patent

Resource Relation:: Patent File Date: 2013 Jun 29

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c.  United States: N. p., 2017. 
        Web.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Grider, Gary, & Torres, Aaron. Parallel file system with metadata distributed across partitioned key-value store c.  United States.

Copy to clipboard


                    Bent, John M., Faibish, Sorin, Grider, Gary, and Torres, Aaron. Tue .  
        "Parallel file system with metadata distributed across partitioned key-value store c".  United States.  https://www.osti.gov/servlets/purl/1393429.

Copy to clipboard


                    
@article{osti_1393429,

  title        = {Parallel file system with metadata distributed across partitioned key-value store c},

  author       = {Bent, John M. and Faibish, Sorin and Grider, Gary and Torres, Aaron},

  abstractNote = {Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).},

  doi          = {},

  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Tue Sep 19 00:00:00 EDT 2017},

  month        = {Tue Sep 19 00:00:00 EDT 2017}

}

Copy to clipboard

Patent:

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

System method and computer program product for dynamically sizing hash tables
patent, September 1999

Schimmel, Curt F.
US Patent Document 5,960,434
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/5960434

File I/O control method
patent-application, July 2002

Sonoda, Kouji; Utsunomiya, Naoki; Kumazaki, Hiroyuki
US Patent Application 10/024336; 20020091751
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20020091751

Method of controlling access to computing resource within shared computing environment
patent, April 2006

Karisson, Magnus; Karamanolis, Christos; Zhu, Xiaoyun
US Patent Application 10/972232; 20060090163
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20060090163

Distributed file system and method of operating a distributed file system
patent-application, May 2006

Tichy, Walter; Isaila, Florin
US Patent Application 10/491459; 20060101025
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20060101025

Method for providing high performance scalable file I/O through persistent file domain and functional partitioning
patent-application, May 2008

Moreira, Jose; Sahoo, Ramendra; Yu, Hao
US Patent Application 11/604162; 20080120435
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20080120435

Switch-Aware Parallel File System
patent-application, December 2011

Hildebrand, Dean; Tewari, Renu
US Patent Application 12/788937; 20110296422
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20110296422

Methods for optimizing data movement in solid state devices
patent-application, June 2013

Givargis, Tony
US Patent Application 13/482865; 20130024644
URL: https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20130024644

Parallel File System Analysis Through Application I/O Tracing
journal, May 2012

Wright, S. A.; Hammond, S. D.; Pennycook, S. J.
The Computer Journal, Vol. 56, Issue 2, p. 141-155
https://doi.org/10.1093/comjnl/bxs044

Similar Records in DOE Patents and OSTI.GOV collections:

Distributed metadata servers for cluster file systems using shared low latency persistent key-value metadata store

Patent Bent, John M.; Faibish, Sorin; Pedone, Jr., James M.; ...

A cluster file system is provided having a plurality of distributed metadata servers with shared access to one or more shared low latency persistent key-value metadata stores. A metadata server comprises an abstract storage interface comprising a software interface module that communicates with at least one shared persistent key-value metadata store providing a key-value interface for persistent storage of key-value metadata. The software interface module provides the key-value metadata to the at least one shared persistent key-value metadata store in a key-value format. The shared persistent key-value metadata store is accessed by a plurality of metadata servers. A metadata requestmore » « less
Full Text Available
Parallel storage system with burst buffer appliance for storage of partitioned key-value store across a plurality of storage tiers

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Parallel storage systems are provided with a burst buffer appliance for storage of a partitioned key-value store across a plurality of storage tiers. The burst buffer appliance is configured to store key-value data on first and second storage tiers comprising respective disjoint subsets of storage. One or more servers in the partitioned key-value store provide (i) an interface between one or more local key-value stores and one or more additional storage architectures, and/or (ii) a key value interface that provides access to at least one hardware key-value store. The local key-values are optionally sorted both locally by the local storemore » « less
Full Text Available
Parallel sort with a ranged, partitioned key-value store in a high perfomance computing environment

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Improved sorting techniques are provided that perform a parallel sort using a ranged, partitioned key-value store in a high performance computing (HPC) environment. A plurality of input data files comprising unsorted key-value data in a partitioned key-value store are sorted. The partitioned key-value store comprises a range server for each of a plurality of ranges. Each input data file has an associated reader thread. Each reader thread reads the unsorted key-value data in the corresponding input data file and performs a local sort of the unsorted key-value data to generate sorted key-value data. A plurality of sorted, ranged subsets ofmore » « less
Full Text Available
Partitioned key-value store with one-sided communications for secondary global key lookup by range-knowledgeable clients

Patent Bent, John M.; Faibish, Sorin; Grider, Gary; ...

Partitioned key-value stores with one-sided communications are provided for secondary global key lookup by range-knowledgeable clients. A secondary global lookups is performed in a partitioned key-value store by receiving a request from an application for a secondary global key; accessing metadata to identify a first server storing the secondary global key; instructing the identified first server to store a first value corresponding to the requested secondary global key into a memory address of the client; and returning program control to the application, wherein the first server identifies a primary key corresponding to the requested secondary global key and instructs amore » « less
Full Text Available
Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata

Patent Bent, John M.; Faibish, Sorin; Grider, Gary

A file system is provided having a plurality of distributed metadata servers with separate metadata servers for file metadata and directory metadata. An exemplary distributed metadata storage system comprises a plurality of compute nodes, wherein at least one compute node comprises a file system client that generates a directory metadata request and/or a file metadata request; a plurality of directory metadata servers to serve directory metadata in response to a given directory metadata request; and a plurality of file metadata servers to serve file metadata in response to a given file metadata request. The plurality of directory metadata servers aremore » « less
Full Text Available

Similar Records

Title: Parallel file system with metadata distributed across partitioned key-value store c

Abstract

Citation Formats

System method and computer program product for dynamically sizing hash tables patent, September 1999

File I/O control method patent-application, July 2002

Method of controlling access to computing resource within shared computing environment patent, April 2006

Distributed file system and method of operating a distributed file system patent-application, May 2006

Method for providing high performance scalable file I/O through persistent file domain and functional partitioning patent-application, May 2008

Switch-Aware Parallel File System patent-application, December 2011

Methods for optimizing data movement in solid state devices patent-application, June 2013

Parallel File System Analysis Through Application I/O Tracing journal, May 2012

System method and computer program product for dynamically sizing hash tables
patent, September 1999

File I/O control method
patent-application, July 2002

Method of controlling access to computing resource within shared computing environment
patent, April 2006

Distributed file system and method of operating a distributed file system
patent-application, May 2006

Method for providing high performance scalable file I/O through persistent file domain and functional partitioning
patent-application, May 2008

Switch-Aware Parallel File System
patent-application, December 2011

Methods for optimizing data movement in solid state devices
patent-application, June 2013

Parallel File System Analysis Through Application I/O Tracing
journal, May 2012