Flexible Data-Aware Scheduling for Workflows over an In-Memory Object Store
This paper explores novel techniques for improving the performance of many-task workflows based on the Swift scripting language. We propose novel programmer options for automated distributed data placement and task scheduling. These options trigger a data placement mechanism used for distributing intermediate workflow data over the servers of Hercules, a distributed key-value store that can be used to cache file system data. We demonstrate that these new mechanisms can significantly improve the aggregated throughput of many-task workflows with up to 86x, reduce the contention on the shared file system, exploit the data locality, and trade off locality and load balance.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science - Office of Advanced Scientific Computing Research; USDOE Office of Science (SC); Spanish Ministerio de Economia y Competitividad (MINECO); European Commission - Community Research and Development Information Service (CORDIS) - Seventh Framework Programme (FP7)
- DOE Contract Number:
- AC02-06CH11357
- OSTI ID:
- 1336153
- Resource Relation:
- Conference: 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 05/16/16 - 05/19/16, Cartagena, CO
- Country of Publication:
- United States
- Language:
- English
Similar Records
Experimental evaluation of a flexible I/O architecture for accelerating workflow engines in ultrascale environments
Data Locality Enhancement of Dynamic Simulations for Exascale Computing (Final Report)
Swift : fast, reliable, loosely coupled parallel computation.
Journal Article
·
Thu Oct 06 00:00:00 EDT 2016
· Parallel Computing
·
OSTI ID:1336153
+3 more
Data Locality Enhancement of Dynamic Simulations for Exascale Computing (Final Report)
Technical Report
·
Fri Nov 29 00:00:00 EST 2019
·
OSTI ID:1336153
Swift : fast, reliable, loosely coupled parallel computation.
Conference
·
Mon Jan 01 00:00:00 EST 2007
·
OSTI ID:1336153
+6 more