AnalyzeThis: An Analysis Workflow-Aware Storage System

Sim, Hyogi; Kim, Youngjae; Vazhkudai, Sudharshan S; Tiwari, Devesh; Anwar, Ali; Butt, Ali R; Ramakrishnan, Lavanya

Title: AnalyzeThis: An Analysis Workflow-Aware Storage System

Full Record
Other Related Research

Abstract

The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. We implement the AnalyzeThis storage system atop an emulation platform of the Active Flash array. Our results indicate that AnalyzeThis is viable, expediting workflow execution and minimizing data movement.

Authors:

Sim, Hyogi ^[1]; Kim, Youngjae ^[1]; Vazhkudai, Sudharshan S ^[1]; Tiwari, Devesh ^[1]; Anwar, Ali ^[2]; Butt, Ali R ^[2]; Ramakrishnan, Lavanya ^[3]

ORNL
Virginia Tech, Blacksburg, VA
Lawrence Berkeley National Laboratory (LBNL)

Publication Date:: Thu Jan 01 00:00:00 EST 2015

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1265619

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: 2015 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC'15), Austin, TX, USA, 20151115, 20151120

Country of Publication:: United States

Language:: English

Subject:: Data Analytics; Visualization & Storage

Citation Formats


                    Sim, Hyogi, Kim, Youngjae, Vazhkudai, Sudharshan S, Tiwari, Devesh, Anwar, Ali, Butt, Ali R, and Ramakrishnan, Lavanya. AnalyzeThis: An Analysis Workflow-Aware Storage System.  United States: N. p., 2015. 
        Web.

Copy to clipboard


                    Sim, Hyogi, Kim, Youngjae, Vazhkudai, Sudharshan S, Tiwari, Devesh, Anwar, Ali, Butt, Ali R, & Ramakrishnan, Lavanya. AnalyzeThis: An Analysis Workflow-Aware Storage System.  United States.

Copy to clipboard


                    Sim, Hyogi, Kim, Youngjae, Vazhkudai, Sudharshan S, Tiwari, Devesh, Anwar, Ali, Butt, Ali R, and Ramakrishnan, Lavanya. 2015.  
        "AnalyzeThis: An Analysis Workflow-Aware Storage System".  United States.  https://www.osti.gov/servlets/purl/1265619.

Copy to clipboard


                    
@article{osti_1265619,

  title        = {AnalyzeThis: An Analysis Workflow-Aware Storage System},

  author       = {Sim, Hyogi and Kim, Youngjae and Vazhkudai, Sudharshan S and Tiwari, Devesh and Anwar, Ali and Butt, Ali R and Ramakrishnan, Lavanya},

  abstractNote = {The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. We implement the AnalyzeThis storage system atop an emulation platform of the Active Flash array. Our results indicate that AnalyzeThis is viable, expediting workflow execution and minimizing data movement.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1265619},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Thu Jan 01 00:00:00 EST 2015},

  month        = {Thu Jan 01 00:00:00 EST 2015}

}

Copy to clipboard

Conference:

View Conference (0.63 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

An Analysis Workflow-Aware Storage System for Multi-Core Active Flash Arrays

Journal Article Sim, Hyogi; Vallee, Geoffroy; Kim, Youngjae; ... - IEEE Transactions on Parallel and Distributed Systems

Here, the need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrainedmore »« less
https://doi.org/10.1109/TPDS.2018.2865471

Full Text Available
AnalyzeThis: an analysis workflow-aware storage system

Conference Sim, Hyogi; Kim, Youngjae; Vazhkudai, Sudharshan; ... - Proceedings of SC15: The International Conference for High Performance Computing, Networking, Storage and Analysis

The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices en-able processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained inmore »« less
https://doi.org/10.1145/2807591.2807622
An Integrated Indexing and Search Service for Distributed File Systems

Journal Article Sim, Hyogi; Khan, Awais; Vazhkudai, Sudharshan; ... - IEEE Transactions on Parallel and Distributed Systems

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this article, we present TagIt, a scalable data management service framework aimed at scientific datasets, which can be integrated into prevalent distributed file system architectures. A key feature of TagIt is a scalable, distributed metadata indexing framework, which facilitates a flexible tagging capability to supportmore »« less
Cited by 10
https://doi.org/10.1109/TPDS.2020.2990656

Full Text Available
Tagit: an integrated indexing and search service for file systems

Conference Sim, Hyogi; Kim, Youngjae; Vazhkudai, Sudharshan; ...

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capabilitymore »« less
https://doi.org/10.1145/3126908.3126929
TagIt: An Integrated Indexing and Search Service for File Systems

Conference Sim, Hyogi; Kim, Youngjae; Vazhkudai, Sudharshan; ...

Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy.In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capability tomore »« less
https://doi.org/10.1145/3126908.3126929

Full Text Available

Similar Records