Cooperative storage of shared files in a parallel computing system with dynamic block size

Title: Cooperative storage of shared files in a parallel computing system with dynamic block size

Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).

Inventors:: Bent, John M.; Faibish, Sorin; Grider, Gary

Issue Date:: 2015-11-10

OSTI Identifier:: 1225724

Assignee:: Los Alamos National Security, LLC LANL

Patent Number(s):: 9,183,211

Application Number:: 13/730,080

Contract Number:: AC52-06NA25396

Resource Relation:: Patent File Date: 2012 Dec 28

Research Org:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org:: USDOE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Full Text
View Full Text

Other works cited in this record:

PLFS: a checkpoint filesystem for parallel applications
conference, January 2009

Bent, John; Gibson, Garth; Grider, Gary
DOI: 10.1145/1654059.1654081

Similar records in DOepatents and OSTI.GOV collections:

Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system

Patent Faibish, Sorin ; Bent, John M ; Tzelnic, Percy ; Grider, Gary ; Torres, Aaron

Techniques are provided for storing files in a parallel computing system using sub-files with semantically meaningful boundaries. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a plurality of sub-files. The method comprises the steps of obtaining a user specification of semantic information related to the file; providing the semantic information as a data structure description to a data formatting library write function; and storing the semantic information related to the file with one or more of themore »« less
Full Text Available February 2015
Storing files in a parallel computing system using list-based index to identify replica files

Patent Faibish, Sorin ; Bent, John M. ; Tzelnic, Percy ; Zhang, Zhenhua ; Grider, Gary

Improved techniques are provided for storing files in a parallel computing system using a list-based index to identify file replicas. A file and at least one replica of the file are stored in one or more storage nodes of the parallel computing system. An index for the file comprises at least one list comprising a pointer to a storage location of the file and a storage location of the at least one replica of the file. The file comprises one or more of a complete file and one or more sub-files. The index may also comprise a checksum value formore »« less
Full Text Available July 2015
Storing files in a parallel computing system based on user-specified parser function

Patent Faibish, Sorin ; Bent, John M ; Tzelnic, Percy ; Grider, Gary ; Manzanares, Adam ; Torres, Aaron

Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser canmore »« less
Full Text Available October 2014
Methods and apparatus for multi-resolution replication of files in a parallel computing system using semantic information

Patent Faibish, Sorin ; Bent, John M. ; Tzelnic, Percy ; Grider, Gary ; Torres, Aaron

Techniques are provided for storing files in a parallel computing system using different resolutions. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a sub-file. The method comprises the steps of obtaining semantic information related to the file; generating a plurality of replicas of the file with different resolutions based on the semantic information; and storing the file and the plurality of replicas of the file in one or more storage nodes of the parallel computing system. Themore »« less
Full Text Available October 2015
Storing files in a parallel computing system based on user or application specification

Patent Faibish, Sorin ; Bent, John M. ; Nick, Jeffrey M. ; Grider, Gary ; Torres, Aaron

Techniques are provided for storing files in a parallel computing system based on a user-specification. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a specification from the distributed application indicating how the plurality of files should be stored; and storing one or more of the plurality of files in one or more storage nodes of a multi-tier storage system based on the specification. The plurality of files comprise a plurality of complete files and/or a plurality of sub-files. The specification can optionally be processed by a daemon executing on onemore »« less
Full Text Available March 2016