WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows
- ORNL
- Duke University
- Columbia University
- University of North Carolina, Chapel Hill, Renaissance Computing Insitute (RENCI)
- University of South California
This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-Analyze-Plan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1842617
- Resource Relation:
- Conference: IEEE CLUSTER 2021 - Portland, Oregon, United States of America - 9/7/2021 4:00:00 AM-9/10/2021 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Scheduling Multilevel Deadline-Constrained Scientific Workflows on Clouds Based on Cost Optimization
Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows
Comparison of Resource Platform Selection Approaches for Scientific Workflows
Journal Article
·
Thu Jan 01 00:00:00 EST 2015
· Scientific Programming
·
OSTI ID:1842617
+2 more
Cloud Bursting with GlideinWMS: Means to satisfy ever increasing computing needs for Scientific Workflows
Journal Article
·
Wed Jan 01 00:00:00 EST 2014
· Journal of Physics. Conference Series
·
OSTI ID:1842617
+3 more
Comparison of Resource Platform Selection Approaches for Scientific Workflows
Conference
·
Fri Mar 05 00:00:00 EST 2010
·
OSTI ID:1842617