Applied Parallel Metadata Indexing
- Los Alamos National Laboratory
The GPFS Archive is parallel archive is a parallel archive used by hundreds of users in the Turquoise collaboration network. It houses 4+ petabytes of data in more than 170 million files. Currently, users must navigate the file system to retrieve their data, requiring them to remember file paths and names. A better solution might allow users to tag data with meaningful labels and searach the archive using standard and user-defined metadata, while maintaining security. last summer, I developed the backend to a tool that adheres to these design goals. The backend works by importing GPFS metadata into a MongoDB cluster, which is then indexed on each attribute. This summer, the author implemented security and developed the user interfae for the search tool. To meet security requirements, each database table is associated with a single user, which only stores records that the user may read, and requires a set of credentials to access. The interface to the search tool is implemented using FUSE (Filesystem in USErspace). FUSE is an intermediate layer that intercepts file system calls and allows the developer to redefine how those calls behave. In the case of this tool, FUSE interfaces with MongoDB to issue queries and populate output. A FUSE implementation is desirable because it allows users to interact with the search tool using commands they are already familiar with. These security and interface additions are essential for a usable product.
- Research Organization:
- Los Alamos National Laboratory (LANL)
- Sponsoring Organization:
- DOE/LANL
- DOE Contract Number:
- AC52-06NA25396
- OSTI ID:
- 1048692
- Report Number(s):
- LA-UR-12-23717
- Country of Publication:
- United States
- Language:
- English
Similar Records
Accessing Data Federations with CVMFS
THE NEW ONLINE METADATA EDITOR FOR GENERATING STRUCTURED METADATA