skip to main content

DOE PAGESDOE PAGES

Title: Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments

The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.
Authors:
 [1] ;  [1] ;  [1] ; ORCiD logo [1] ;  [2] ;  [2]
  1. Univ. Carlos III, Madrid (Spain)
  2. Argonne National Lab. (ANL), Lemont, IL (United States)
Publication Date:
Grant/Contract Number:
AC02-06CH11357
Type:
Accepted Manuscript
Journal Name:
Parallel Computing
Additional Journal Information:
Journal Volume: 61; Journal Issue: C; Journal ID: ISSN 0167-8191
Publisher:
Elsevier
Research Org:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org:
Spanish Ministerio de Economia y Competitividad (MINECO); European Commission - Community Research and Development Information Service (CORDIS) - Seventh Framework Programme (FP7); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; cloud computing; high-performance computing; I/O acceleration; workflow
OSTI Identifier:
1364635