Cooperative storage of shared files in a parallel computing system with dynamic block size
Abstract
Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1225724
- Patent Number(s):
- 9183211
- Application Number:
- 13/730,080
- Assignee:
- Los Alamos National Security, LLC
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Resource Relation:
- Patent File Date: 2012 Dec 28
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Bent, John M., Faibish, Sorin, and Grider, Gary. Cooperative storage of shared files in a parallel computing system with dynamic block size. United States: N. p., 2015.
Web.
Bent, John M., Faibish, Sorin, & Grider, Gary. Cooperative storage of shared files in a parallel computing system with dynamic block size. United States.
Bent, John M., Faibish, Sorin, and Grider, Gary. Tue .
"Cooperative storage of shared files in a parallel computing system with dynamic block size". United States. https://www.osti.gov/servlets/purl/1225724.
@article{osti_1225724,
title = {Cooperative storage of shared files in a parallel computing system with dynamic block size},
author = {Bent, John M. and Faibish, Sorin and Grider, Gary},
abstractNote = {Improved techniques are provided for parallel writing of data to a shared object in a parallel computing system. A method is provided for storing data generated by a plurality of parallel processes to a shared object in a parallel computing system. The method is performed by at least one of the processes and comprises: dynamically determining a block size for storing the data; exchanging a determined amount of the data with at least one additional process to achieve a block of the data having the dynamically determined block size; and writing the block of the data having the dynamically determined block size to a file system. The determined block size comprises, e.g., a total amount of the data to be stored divided by the number of parallel processes. The file system comprises, for example, a log structured virtual parallel file system, such as a Parallel Log-Structured File System (PLFS).},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Nov 10 00:00:00 EST 2015},
month = {Tue Nov 10 00:00:00 EST 2015}
}
Works referenced in this record:
Parallel file system and method with quota allocation
patent, August 1999
- Schmuck, Frank B.; Wyllie, James C.; Curran, Robert J.
- US Patent Document 5,946,686
Method and system for providing an interleaved backup
patent, February 2004
- McNeil, Daniel D.; DiMartino, Jorge; Kohli, Jaspal
- US Patent Document 6,691,212
Computer system and process for transferring streams of data between multiple storage units and multiple applications in a scalable and reliable manner
patent, August 2004
- Peters, Eric C.; Rabinowitz, Stanley; Jacobs, Herbert R.
- US Patent Document 6,785,768
Parallel Log Structured File System Collective Buffering to Achieve a Compact Representation of Scientific and/or Dimensional Data
patent-application, June 2013
- Grider, Gary A.; Poole, Stephen W.
- US Patent Document 13/722946; 20130159364
PLFS: a checkpoint filesystem for parallel applications
conference, January 2009
- Bent, John; Gibson, Garth; Grider, Gary