Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system
Techniques are provided for storing files in a parallel computing system using sub-files with semantically meaningful boundaries. A method is provided for storing at least one file generated by a distributed application in a parallel computing system. The file comprises one or more of a complete file and a plurality of sub-files. The method comprises the steps of obtaining a user specification of semantic information related to the file; providing the semantic information as a data structure description to a data formatting library write function; and storing the semantic information related to the file with one or more of the sub-files in one or more storage nodes of the parallel computing system. The semantic information provides a description of data in the file. The sub-files can be replicated based on semantically meaningful boundaries.
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- Assignee:
- EMC Corporation (Hopkinton, MA)
- Patent Number(s):
- 8,949,255
- Application Number:
- 13/536,384
- OSTI ID:
- 1169058
- Country of Publication:
- United States
- Language:
- English
Systems and methods for managing portions of files in multi-tier storage systems
|
patent | January 2013 |
Rule based aggregation of files and transactions in a switched file system
|
patent-application | July 2004 |
| PLFS: a checkpoint filesystem for parallel applications | conference | January 2009 |
Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system
|
patent | February 2015 |
Similar Records
Storing files in a parallel computing system based on user-specified parser function
Storing files in a parallel computing system using list-based index to identify replica files