skip to main content

DOE PAGESDOE PAGES

Title: Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments

The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.
Authors:
 [1] ;  [1] ;  [1] ; ORCiD logo [1] ;  [2] ;  [2]
  1. Univ. Carlos III, Madrid (Spain)
  2. Argonne National Lab. (ANL), Lemont, IL (United States)
Publication Date:
Grant/Contract Number:
AC02-06CH11357
Type:
Accepted Manuscript
Journal Name:
Parallel Computing
Additional Journal Information:
Journal Volume: 61; Journal Issue: C; Journal ID: ISSN 0167-8191
Publisher:
Elsevier
Research Org:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org:
Spanish Ministerio de Economia y Competitividad (MINECO); European Commission - Community Research and Development Information Service (CORDIS) - Seventh Framework Programme (FP7); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; cloud computing; high-performance computing; I/O acceleration; workflow
OSTI Identifier:
1364635

Duro, Francisco Rodrigo, Blas, Javier Garcia, Isaila, Florin, Carretero, Jesus, Wozniak, Justin M., and Ross, Rob. Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments. United States: N. p., Web. doi:10.1016/j.parco.2016.10.003.
Duro, Francisco Rodrigo, Blas, Javier Garcia, Isaila, Florin, Carretero, Jesus, Wozniak, Justin M., & Ross, Rob. Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments. United States. doi:10.1016/j.parco.2016.10.003.
Duro, Francisco Rodrigo, Blas, Javier Garcia, Isaila, Florin, Carretero, Jesus, Wozniak, Justin M., and Ross, Rob. 2016. "Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments". United States. doi:10.1016/j.parco.2016.10.003. https://www.osti.gov/servlets/purl/1364635.
@article{osti_1364635,
title = {Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments},
author = {Duro, Francisco Rodrigo and Blas, Javier Garcia and Isaila, Florin and Carretero, Jesus and Wozniak, Justin M. and Ross, Rob},
abstractNote = {The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.},
doi = {10.1016/j.parco.2016.10.003},
journal = {Parallel Computing},
number = C,
volume = 61,
place = {United States},
year = {2016},
month = {10}
}