skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: A Methodology for the Rapid Development of Scalable HPC Data Services

Abstract

There is growing evidence in the scientifc computing community that parallel file systems are not sufficient for all HPC storage workloads. This realization has motivated extensive research in new storage system designs. The question of which alternative we should turn towards implies that there could be a single answer satisfying a wide range of very diverse applications. We argue that such a generic solution does not exist. Instead, custom data services should be designed and tailored to the needs of specific applications on specific hardware. Furthermore, custom data services should be designed in close collaboration with users. In this paper, we present a methodology for the rapid development of such data services. This methodology promotes the design of reusable building blocks that can be composed together efficiently through a runtime based on high-performance threading, tasking, and remote procedure calls. We illustrate the success of our methodology by showcasing three examples of data services designed from the same building blocks, yet targeting entirely different applications.

Authors:
 [1];  [1];  [1];  [1];  [1];  [1];  [1];  [2];  [2];  [2];  [2]; ;  [3]; ORCiD logo [4];  [3]
  1. Argonne
  2. Los Alamos
  3. Fermilab
  4. Fermmilab
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States); Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1490839
Report Number(s):
FERMILAB-CONF-18-575-CD
1712343
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Conference
Resource Relation:
Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Dallas, TX, USA, 11/11-11/16/2018
Country of Publication:
United States
Language:
English

Citation Formats

Dorier, Mattieu, Carns, Philip, Harms, Kevin, Latham, Robert, Ross, Robert, Snyder, Shane, Wozniak, Justin, Gutierrez, Samuel K., Robey, Bob, Settlemyer, Brad, Shipman, Galen, Soumagne, Jerome, Kowalkowski, James, Paterno, Marc, and Sehrish, Saba. A Methodology for the Rapid Development of Scalable HPC Data Services. United States: N. p., 2018. Web.
Dorier, Mattieu, Carns, Philip, Harms, Kevin, Latham, Robert, Ross, Robert, Snyder, Shane, Wozniak, Justin, Gutierrez, Samuel K., Robey, Bob, Settlemyer, Brad, Shipman, Galen, Soumagne, Jerome, Kowalkowski, James, Paterno, Marc, & Sehrish, Saba. A Methodology for the Rapid Development of Scalable HPC Data Services. United States.
Dorier, Mattieu, Carns, Philip, Harms, Kevin, Latham, Robert, Ross, Robert, Snyder, Shane, Wozniak, Justin, Gutierrez, Samuel K., Robey, Bob, Settlemyer, Brad, Shipman, Galen, Soumagne, Jerome, Kowalkowski, James, Paterno, Marc, and Sehrish, Saba. Tue . "A Methodology for the Rapid Development of Scalable HPC Data Services". United States. https://www.osti.gov/servlets/purl/1490839.
@article{osti_1490839,
title = {A Methodology for the Rapid Development of Scalable HPC Data Services},
author = {Dorier, Mattieu and Carns, Philip and Harms, Kevin and Latham, Robert and Ross, Robert and Snyder, Shane and Wozniak, Justin and Gutierrez, Samuel K. and Robey, Bob and Settlemyer, Brad and Shipman, Galen and Soumagne, Jerome and Kowalkowski, James and Paterno, Marc and Sehrish, Saba},
abstractNote = {There is growing evidence in the scientifc computing community that parallel file systems are not sufficient for all HPC storage workloads. This realization has motivated extensive research in new storage system designs. The question of which alternative we should turn towards implies that there could be a single answer satisfying a wide range of very diverse applications. We argue that such a generic solution does not exist. Instead, custom data services should be designed and tailored to the needs of specific applications on specific hardware. Furthermore, custom data services should be designed in close collaboration with users. In this paper, we present a methodology for the rapid development of such data services. This methodology promotes the design of reusable building blocks that can be composed together efficiently through a runtime based on high-performance threading, tasking, and remote procedure calls. We illustrate the success of our methodology by showcasing three examples of data services designed from the same building blocks, yet targeting entirely different applications.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {10}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: