Storing files in a parallel computing system based on user-specified parser function
Techniques are provided for storing files in a parallel computing system based on a user-specified parser function. A plurality of files generated by a distributed application in a parallel computing system are stored by obtaining a parser from the distributed application for processing the plurality of files prior to storage; and storing one or more of the plurality of files in one or more storage nodes of the parallel computing system based on the processing by the parser. The plurality of files comprise one or more of a plurality of complete files and a plurality of sub-files. The parser can optionally store only those files that satisfy one or more semantic requirements of the parser. The parser can also extract metadata from one or more of the files and the extracted metadata can be stored with one or more of the plurality of files and used for searching for files.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- Assignee:
- EMC Corporation (Hopkinton, MA)
- Patent Number(s):
- 8,868,576
- Application Number:
- 13/536,369
- OSTI ID:
- 1160235
- Resource Relation:
- Patent File Date: 2012 Jun 28
- Country of Publication:
- United States
- Language:
- English
Method and system for data transfer between compute clusters and file system
|
patent | April 2017 |
Multi-tier caching
|
patent | May 2016 |
Architecture and method for a burst buffer using flash technology
|
patent | March 2016 |
Similar Records
Parallel file system with metadata distributed across partitioned key-value store c
Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system