Parallel file system with metadata distributed across partitioned key-value store c
Improved techniques are provided for storing metadata associated with a plurality of sub-files associated with a single shared file in a parallel file system. The shared file is generated by a plurality of applications executing on a plurality of compute nodes. A compute node implements a Parallel Log Structured File System (PLFS) library to store at least one portion of the shared file generated by an application executing on the compute node and metadata for the at least one portion of the shared file on one or more object storage servers. The compute node is also configured to implement a partitioned data store for storing a partition of the metadata for the shared file, wherein the partitioned data store communicates with partitioned data stores on other compute nodes using a message passing interface. The partitioned data store can be implemented, for example, using Multidimensional Data Hashing Indexing Middleware (MDHIM).
- Research Organization:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-06NA25396
- Assignee:
- EMC IP Holding Company LLC
- Patent Number(s):
- 9,767,107
- Application Number:
- 13/931,832
- OSTI ID:
- 1393429
- Resource Relation:
- Patent File Date: 2013 Jun 29
- Country of Publication:
- United States
- Language:
- English
Similar Records
Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata
Parallel storage system with burst buffer appliance for storage of partitioned key-value store across a plurality of storage tiers