Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments
The increasing volume of scientific data and the limited scalability and performance of storage systems are currently presenting a significant limitation for the productivity of the scientific workflows running on both high-performance computing (HPC) and cloud platforms. Clearly needed is better integration of storage systems and workflow engines to address this problem. This paper presents and evaluates a novel solution that leverages codesign principles for integrating Hercules—an in-memory data store—with a workflow management system. We consider four main aspects: workflow representation, task scheduling, task placement, and task termination. As a result, the experimental evaluation on both cloud and HPC systems demonstrates significant performance and scalability improvements over existing state-of-the-art approaches.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- Spanish Ministerio de Economia y Competitividad (MINECO); European Commission - Community Research and Development Information Service (CORDIS) - Seventh Framework Programme (FP7); USDOE Office of Science (SC), Basic Energy Sciences (BES)
- Grant/Contract Number:
- AC02-06CH11357
- OSTI ID:
- 1364635
- Alternate ID(s):
- OSTI ID: 1572592
- Journal Information:
- Parallel Computing, Vol. 61, Issue C; ISSN 0167-8191
- Publisher:
- ElsevierCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Web of Science
Similar Records
Accelerating Scientific Workflows on HPC Platforms with In Situ Processing
MaDaTS: Managing Data on Tiered Storage for Scientific Workflows