TagIt: An Integrated Indexing and Search Service for File Systems

Conference · Wed Nov 01 00:00:00 EDT 2017

DOI:https://doi.org/10.1145/3126908.3126929· OSTI ID:1423035

Sim, Hyogi  ^[1]; Kim, Youngjae ^[2]; Vazhkudai, Sudharshan  ^[1]; Vallee, Geoffroy  ^[1]; Lim, Seung-Hwan  ^[1]; Butt, Ali R. ^[3]

---

1. ORNL
2. Sogang University
3. Virginia Tech, Blacksburg, VA

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy.In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capability to support data discovery. The tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. Our evaluation shows that TagIt can expedite data search by up to 10X over the extant decoupled approach.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1423035
• Resource Relation: Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis - Denver, Colorado, United States of America - 11/12/2017 3:00:00 PM-11/17/2017 3:00:00 PM
• Country of Publication: United States
• Language: English