skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Could Blobs Fuel Storage-Based Convergence between HPC and Big Data?

Abstract

The increasingly growing data sets processed on HPC platforms raise major challenges for the underlying storage layer. A promising alternative to POSIX-IO- compliant file systems are simpler blobs (binary large objects), or object storage systems. Such systems offer lower overhead and better performance at the cost of largely unused features such as file hierarchies or permissions. Similarly, blobs are increasingly considered for replacing distributed file systems for big data analytics or as a base for storage abstractions such as key-value stores or time-series databases. This growing interest in such object storage on HPC and big data platforms raises the question: Are blobs the right level of abstraction to enable storage-based convergence between HPC and Big Data? In this paper we study the impact of blob-based storage for real-world applications on HPC and cloud environments. The results show that blobbased storage convergence is possible, leading to a significant performance improvement on both platforms

Authors:
; ; ; ; ;
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
European Commission (EC); Agence Nationale de la recherche (ANR); USDOE Office of Science - Office of Advanced Scientific Computing Research
OSTI Identifier:
1392626
DOE Contract Number:  
AC02-06CH11357
Resource Type:
Conference
Resource Relation:
Conference: 2017 IEEE CLUSTER , 09/05/17 - 09/08/17, Waikiki, HI, US
Country of Publication:
United States
Language:
English

Citation Formats

Matri, Pierre, Alforov, Yevhen, Brandon, Alvaro, Kuhn, Michael, Carns, Philip, and Ludwig, Thomas. Could Blobs Fuel Storage-Based Convergence between HPC and Big Data?. United States: N. p., 2017. Web.
Matri, Pierre, Alforov, Yevhen, Brandon, Alvaro, Kuhn, Michael, Carns, Philip, & Ludwig, Thomas. Could Blobs Fuel Storage-Based Convergence between HPC and Big Data?. United States.
Matri, Pierre, Alforov, Yevhen, Brandon, Alvaro, Kuhn, Michael, Carns, Philip, and Ludwig, Thomas. Tue . "Could Blobs Fuel Storage-Based Convergence between HPC and Big Data?". United States. doi:. https://www.osti.gov/servlets/purl/1392626.
@article{osti_1392626,
title = {Could Blobs Fuel Storage-Based Convergence between HPC and Big Data?},
author = {Matri, Pierre and Alforov, Yevhen and Brandon, Alvaro and Kuhn, Michael and Carns, Philip and Ludwig, Thomas},
abstractNote = {The increasingly growing data sets processed on HPC platforms raise major challenges for the underlying storage layer. A promising alternative to POSIX-IO- compliant file systems are simpler blobs (binary large objects), or object storage systems. Such systems offer lower overhead and better performance at the cost of largely unused features such as file hierarchies or permissions. Similarly, blobs are increasingly considered for replacing distributed file systems for big data analytics or as a base for storage abstractions such as key-value stores or time-series databases. This growing interest in such object storage on HPC and big data platforms raises the question: Are blobs the right level of abstraction to enable storage-based convergence between HPC and Big Data? In this paper we study the impact of blob-based storage for real-world applications on HPC and cloud environments. The results show that blobbased storage convergence is possible, leading to a significant performance improvement on both platforms},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Sep 05 00:00:00 EDT 2017},
month = {Tue Sep 05 00:00:00 EDT 2017}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: