skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Abstract

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.

Authors:
; ; ; ; ;
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
National Science Foundation (NSF); USDOE Office of Science - Office of Advanced Scientific Computing Research
OSTI Identifier:
1523625
DOE Contract Number:  
AC02-06CH11357
Resource Type:
Conference
Resource Relation:
Conference: 38th IEEE International Conference on Distributed Computing Systems, 07/02/18 - 07/05/18, Vienna, AT
Country of Publication:
United States
Language:
English
Subject:
Big Data convergence; HPC developers; SLoG design; big data; centralized parallel file system; cloud computing; cloud developers; computational modeling; concurrent computing; convergence; data structure; data structures; file systems; hpc; i/o; large-scale logging middleware; latency; middleware; parallel file system; parallel processing; replication; shared log abstraction; shared log middleware; small files; storage; storage model; supercomputers; telemetry; throughput

Citation Formats

Matri, Pierre, Carns, Philip, Ross, Robert, Costan, Alexandru, Perez, Maria S., and Antoniu, Gabriel. SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence. United States: N. p., 2018. Web. doi:10.1109/ICDCS.2018.00156.
Matri, Pierre, Carns, Philip, Ross, Robert, Costan, Alexandru, Perez, Maria S., & Antoniu, Gabriel. SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence. United States. doi:10.1109/ICDCS.2018.00156.
Matri, Pierre, Carns, Philip, Ross, Robert, Costan, Alexandru, Perez, Maria S., and Antoniu, Gabriel. Mon . "SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence". United States. doi:10.1109/ICDCS.2018.00156.
@article{osti_1523625,
title = {SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence},
author = {Matri, Pierre and Carns, Philip and Ross, Robert and Costan, Alexandru and Perez, Maria S. and Antoniu, Gabriel},
abstractNote = {Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG’s design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.},
doi = {10.1109/ICDCS.2018.00156},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {7}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: