WIRE: Resource-efficient Scaling with Online Prediction for DAG-based Workflows

Conference · Fri Oct 01 00:00:00 EDT 2021

DOI:https://doi.org/10.1109/Cluster48925.2021.00025· OSTI ID:1842617

Xie, Bing ^[1]; Cao, Qiang ^[2]; Kunjir, Mayuresh ^[2]; Wan, Linli ^[3]; Chase, Jeffrey ^[2]; Mandal, Anirban ^[4]; Rynge, Mats ^[5]

---

1. ORNL
2. Duke University
3. Columbia University
4. University of North Carolina, Chapel Hill, Renaissance Computing Insitute (RENCI)
5. University of South California

This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-Analyze-Plan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1842617
• Resource Relation: Conference: IEEE CLUSTER 2021 - Portland, Oregon, United States of America - 9/7/2021 4:00:00 AM-9/10/2021 4:00:00 AM
• Country of Publication: United States
• Language: English