Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Wang, Ke; Qiao, Kan; Sadooghi, Iman; Zhou, Xiaobing; Li, Tonglin; Lang, Michael; Raicu, Ioan

doi:10.1002/cpe.3617

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Journal Article · 14 August 2015 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.3617· OSTI ID:1786148

Wang, Ke ^[1]; Qiao, Kan ^[2]; Sadooghi, Iman ^[1]; Zhou, Xiaobing ^[3]; Li, Tonglin ^[1]; Lang, Michael ^[4]; Raicu, Ioan ^[5]

Department of Computer Science Illinois Institute of Technology 10 W 31st St, Stuart Building, Room 002 Chicago IL 60616 USA
Google Inc. Seattle WA 98103 USA
Hortonworks Inc. Santa Clara CA USA
Los Alamos National Laboratory Los Alamos NM USA
Department of Computer Science Illinois Institute of Technology 10 W 31st St, Stuart Building, Room 002 Chicago IL 60616 USA, Argonne National Laboratory Lemont IL USA

Summary

Data‐driven programming models such as many‐task computing (MTC) have been prevalent for running data‐intensive scientific applications. MTC applies over‐decomposition to enable distributed scheduling. To achieve extreme scalability, MTC proposes a fully distributed task scheduling architecture that employs as many schedulers as the compute nodes to make scheduling decisions. Achieving distributed load balancing and best exploiting data locality are two important goals for the best performance of distributed scheduling of data‐intensive applications. Our previous research proposed a data‐aware work‐stealing technique to optimize both load balancing and data locality by using both dedicated and shared task ready queues in each scheduler. Tasks were organized in queues based on the input data size and location. Distributed key‐value store was applied to manage task metadata. We implemented the technique in MATRIX, a distributed MTC task execution framework. In this work, we devise an analytical suboptimal upper bound of the proposed technique, compare MATRIX with other scheduling systems, and explore the scalability of the technique at extreme scales. Results show that the technique is not only scalable but can achieve performance within 15% of the suboptimal solution. Copyright © 2015 John Wiley & Sons, Ltd.

View Accepted Manuscript (Publisher)

Sponsoring Organization:: USDOE

Grant/Contract Number:: FC02-06ER25750

OSTI ID:: 1786148

Journal Information:: Concurrency and Computation. Practice and Experience, Journal Name: Concurrency and Computation. Practice and Experience Journal Issue: 1 Vol. 28; ISSN 1532-0626

Publisher:: Wiley Blackwell (John Wiley & Sons)Copyright Statement

Country of Publication:: United Kingdom

Language:: English

References (41)

Distributed computing in practice: the Condor experience Thain, Douglas; Tannenbaum, Todd; Livny, Miron Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356 https://doi.org/10.1002/cpe.938	journal	January 2005
Scalable Load Balancing Techniques for Parallel Computers Kumar, V.; Grama, A. Y.; Vempaty, N. R. Journal of Parallel and Distributed Computing, Vol. 22, Issue 1 https://doi.org/10.1006/jpdc.1994.1070	journal	July 1994
Impact of Over-Decomposition on Coordinated Checkpoint/Rollback Protocol Besseron, Xavier; Gautier, Thierry Euro-Par 2011: Parallel Processing Workshops https://doi.org/10.1007/978-3-642-29740-3_36	book	January 2012
NP-complete scheduling problems Ullman, J. D. Journal of Computer and System Sciences, Vol. 10, Issue 3 https://doi.org/10.1016/S0022-0000(75)80008-0	journal	June 1975
Strategies for dynamic load balancing on highly parallel computers Willebeek-LeMair, M. H.; Reeves, A. P. IEEE Transactions on Parallel and Distributed Systems, Vol. 4, Issue 9 https://doi.org/10.1109/71.243526	journal	January 1993
An approximate analysis of the join the shortest queue (JSQ) policy Lin, Hwa-Chun; Raghavendra, C. S. IEEE Transactions on Parallel and Distributed Systems, Vol. 7, Issue 3, p. 301-307 https://doi.org/10.1109/71.491583	journal	March 1996
FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems Zhao, Dongfang; Zhang, Zhao; Zhou, Xiaobing 2014 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2014.7004214	conference	October 2014
Optimizing load balancing and data-locality with data-aware scheduling Wang, Ke; Zhou, Xraobing; Li, Tonglin 2014 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2014.7004220	conference	October 2014
Achieving Efficient Distributed Scheduling with Message Queues in the Cloud for Many-Task Computing and High-Performance Computing Sadooghi, Iman; Palur, Sandeep; Anthony, Ajay 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2014.30	conference	May 2014
Overcoming Hadoop Scaling Limitations through Distributed Task Execution Wang, Ke; Liu, Ning; Sadooghi, Iman 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.42	conference	September 2015
GRAPH/Z: A Key-Value Store Based Scalable Graph Processing System Li, Tonglin; Ma, Chaoqi; Li, Jiabao 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.90	conference	September 2015
On the Merits of Distributed Work-Stealing on Selective Locality-Aware Tasks Paudel, Jeeva; Tardieu, Olivier; Amaral, Jose Nelson 2013 42nd International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2013.19	conference	October 2013
Hierarchical Load Balancing for Charm++ Applications on Large Supercomputers Zheng, Gengbin; Meneses, Esteban; Bhatele, Abhinav 2010 International Conference on Parallel Processing Workshops (ICPPW), 2010 39th International Conference on Parallel Processing Workshops https://doi.org/10.1109/ICPPW.2010.65	conference	September 2010
ZHT: A Light-Weight Reliable Persistent Dynamic Scalable Zero-Hop Distributed Hash Table Li, Tonglin; Zhou, Xiaobing; Brandstatter, Kevin 2013 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), 2013 IEEE 27th International Symposium on Parallel and Distributed Processing https://doi.org/10.1109/IPDPS.2013.110	conference	May 2013
Modeling Many-Task Computing Workloads on a Petaflop IBM Blue Gene/P Supercomputer Wang, Ke; Ma, Zhangjie; Raicu, Ioan 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW) https://doi.org/10.1109/IPDPSW.2013.274	conference	May 2013
The Hadoop Distributed File System Shvachko, Konstantin; Kuang, Hairong; Radia, Sanjay 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2010.5496972	conference	May 2010
Many-task computing for grids and supercomputers Raicu, Ioan; Foster, Ian T. 2008 Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS) https://doi.org/10.1109/MTAGS.2008.4777912	conference	November 2008
Toward loosely coupled programming on petascale systems Raicu, Ioan; Wilde, Mike 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2008.5219768	conference	November 2008
Optimizing Data Locality for Fork/Join Programs Using Constrained Work Stealing Lifflander, Jonathan; Krishnamoorthy, Sriram; Kale, Laxmikant V. SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.75	conference	November 2014
Swift: Fast, Reliable, Loosely Coupled Parallel Computation Zhao, Yong; Hategan, Mihael; Clifford, Ben 2007 IEEE Congress on Services (Services 2007) https://doi.org/10.1109/SERVICES.2007.63	conference	July 2007
All-Pairs: An Abstraction for Data-Intensive Computing on Campus Grids Moretti, Christopher; Bui, Hoang; Hollingsworth, Karen IEEE Transactions on Parallel and Distributed Systems, Vol. 21, Issue 1 https://doi.org/10.1109/TPDS.2009.49	journal	January 2010
Dynamic circular work-stealing deque Chase, David; Lev, Yossi Proceedings of the 17th annual ACM symposium on Parallelism in algorithms and architectures - SPAA'05 https://doi.org/10.1145/1073970.1073974	conference	January 2005
X10: an object-oriented approach to non-uniform cluster computing Charles, Philippe; Grothoff, Christian; Saraswat, Vijay Proceedings of the 20th annual ACM SIGPLAN conference on Object oriented programming systems languages and applications - OOPSLA '05 https://doi.org/10.1145/1094811.1094852	conference	January 2005
X10: an object-oriented approach to non-uniform cluster computing Charles, Philippe; Grothoff, Christian; Saraswat, Vijay ACM SIGPLAN Notices, Vol. 40, Issue 10 https://doi.org/10.1145/1103845.1094852	journal	October 2005
Dryad: distributed data-parallel programs from sequential building blocks Isard, Michael; Budiu, Mihai; Yu, Yuan Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 - EuroSys '07 https://doi.org/10.1145/1272996.1273005	conference	January 2007
Dynamo: amazon's highly available key-value store DeCandia, Giuseppe; Hastorun, Deniz; Jampani, Madan Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles - SOSP '07 https://doi.org/10.1145/1294261.1294281	conference	January 2007
Falkon: a Fast and Light-weight tasK executiON framework Raicu, Ioan; Zhao, Yong; Dumitrescu, Catalin Proceedings of the 2007 ACM/IEEE conference on Supercomputing - SC '07 https://doi.org/10.1145/1362622.1362680	conference	January 2007
Accelerating large-scale data exploration through data diffusion Raicu, Ioan; Zhao, Yong; Foster, Ian T. Proceedings of the 2008 international workshop on Data-aware distributed computing - DADC '08 https://doi.org/10.1145/1383519.1383521	conference	January 2008
Analysis of size interval task assignment policies Bachmat, Eitan; Sarfati, Hagit ACM SIGMETRICS Performance Evaluation Review, Vol. 36, Issue 2 https://doi.org/10.1145/1453175.1453199	journal	August 2008
The quest for scalable support of data-intensive workloads in distributed systems Raicu, Ioan; Foster, Ian T.; Zhao, Yong Proceedings of the 18th ACM international symposium on High performance distributed computing - HPDC '09 https://doi.org/10.1145/1551609.1551642	conference	January 2009
Scalable work stealing Dinan, James; Larkins, D. Brian; Sadayappan, P. Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09 https://doi.org/10.1145/1654059.1654113	conference	January 2009
SLAW: a scalable locality-aware adaptive work-stealing scheduler for multi-core systems Guo, Yi; Zhao, Jisheng; Cave, Vincent Proceedings of the 15th ACM SIGPLAN symposium on Principles and practice of parallel programming - PPoPP '10 https://doi.org/10.1145/1693453.1693504	conference	January 2010
Using simulation to explore distributed key-value stores for extreme-scale system services Wang, Ke; Kulkarni, Abhishek; Lang, Michael Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503239	conference	January 2013
A distributed dynamic load balancer for iterative applications Menon, Harshitha; Kalé, Laxmikant SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/2503210.2503284	conference	November 2013
Next generation job management systems for extreme-scale ensemble computing Wang, Ke; Zhou, Xiaobing; Chen, Hao Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600703	conference	January 2014
Towards Scalable Distributed Workload Manager with Monitoring-Based Weakly Consistent Resource Stealing Wang, Ke; Zhou, Xiaobing; Qiao, Kan HPDC'15: The 24th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/2749246.2749249	conference	June 2015
A Dynamically Scalable Cloud Data Infrastructure for Sensor Networks Li, Tonglin; Keahey, Kate; Wang, Ke HPDC'15: The 24th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 6th Workshop on Scientific Cloud Computing https://doi.org/10.1145/2755644.2755650	conference	June 2015
The implementation of the Cilk-5 multithreaded language Frigo, Matteo; Leiserson, Charles E.; Randall, Keith H. ACM SIGPLAN Notices, Vol. 33, Issue 5 https://doi.org/10.1145/277652.277725	journal	May 1998
Epidemic algorithms for replicated database maintenance Demers, Alan; Greene, Dan; Hauser, Carl Proceedings of the sixth annual ACM Symposium on Principles of distributed computing - PODC '87 https://doi.org/10.1145/41840.41841	conference	January 1987
Job placement with unknown duration and no preemption Harchol-Balter, Mor ACM SIGMETRICS Performance Evaluation Review, Vol. 28, Issue 4 https://doi.org/10.1145/544397.544399	journal	March 2001
A multi-level load balancing scheme for OR-parallel exhaustive search programs on the multi-PSI Furuichi, M.; Taki, K.; Ichiyoshi, N. Proceedings of the second ACM SIGPLAN symposium on Principles & practice of parallel programming - PPOPP '90 https://doi.org/10.1145/99163.99170	conference	January 1990

Similar Records

Locality-aware and load-balanced static task scheduling for MapReduce

Journal Article · 2018 · Future Generations Computer Systems · OSTI ID:1580353

Flexible Data-Aware Scheduling for Workflows over an In-Memory Object Store

Conference · 2015 · OSTI ID:1336153

Load‐balanced and locality‐aware scheduling for data‐intensive workloads at extreme scales

Citation Formats

References (41)

Similar Records

Related Subjects