Alternative mixed integer linear programming optimization for joint job scheduling and data allocation in grid computing

Feng, Shengyu; Kim, Jaehyung; Yang, Yiming; Boudreau, Joseph; Chowdhury, Tasnuva; Hoisie, Adolfy; Khan, Raees; Kilic, Ozgur O.; Klasky, Scott; Korchuganova, Tatiana; Nilsson, Paul; Martinez Outschoorn, Verena Ingrid; Park, David K.; Podhorszki, Norbert; Ren, Yihui; Suter, Frédéric; Sri Vatsavai, Sairam; Yang, Wei; Yoo, Shinjae; Maeno, Tadashi; Klimentov, Alexei

doi:10.1016/j.future.2025.108075

Alternative mixed integer linear programming optimization for joint job scheduling and data allocation in grid computing

Journal Article · Mon Aug 18 00:00:00 EDT 2025 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2025.108075· OSTI ID:2997539

^[1]; ^[2]; ^[1]; ^[3]; ^[4]; ^[4]; ^[3]; ^[4]; ^[5]; ^[3]; ^[4]; ^[6]; ^[4]; ^[5]; ^[4]; ^[4]; ^[4]; ^[7]; ^[4]; ^[4] more »

Carnegie Mellon Univ., Pittsburgh, PA (United States)
Carnegie Mellon Univ., Pittsburgh, PA (United States); Yonsei Univ., Seoul (Korea, Republic of)
Univ. of Pittsburgh, PA (United States)
Brookhaven National Laboratory (BNL), Upton, NY (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Univ. of Massachusetts, Amherst, MA (United States)
SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)

This paper presents a novel approach to the joint optimization of job scheduling and data allocation in grid computing environments. We formulate this joint optimization problem as a mixed integer quadratically constrained program. To tackle the nonlinearity in the constraint, we alternatively fix a subset of decision variables and optimize the remaining ones via Mixed Integer Linear Programming (MILP). We solve the MILP problem at each iteration via an off-the-shelf MILP solver. Our experimental results show that our method significantly outperforms existing heuristic methods, employing either independent optimization or joint optimization strategies. We have also verified the generalization ability of our method over grid environments with various sizes and its high robustness to the algorithm setting.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-00OR22725; SC0012704

OSTI ID:: 2997539

Alternate ID(s):: OSTI ID: 3002213

Report Number(s):: BNL--228973-2025-JAAM

Journal Information:: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Vol. 175; ISSN 0167-739X

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (34)

Genetic algorithm in finding Pareto frontier of optimizing data transfer versus job execution in grids Taheri, Javid; Zomaya, Albert Y.; Khan, Samee U. Concurrency and Computation: Practice and Experience, Vol. 28, Issue 6 https://doi.org/10.1002/cpe.2960	journal	November 2012
Evolutionary Algorithms Bartz‐Beielstein, Thomas; Branke, Jürgen; Mehnen, Jörn WIREs Data Mining and Knowledge Discovery, Vol. 4, Issue 3 https://doi.org/10.1002/widm.1124	journal	April 2014
A Comparison of Eleven Static Heuristics for Mapping a Class of Independent Tasks onto Heterogeneous Distributed Computing Systems Braun, Tracy D.; Siegel, Howard Jay; Beck, Noah Journal of Parallel and Distributed Computing, Vol. 61, Issue 6 https://doi.org/10.1006/jpdc.2000.1714	journal	June 2001
Coordinate descent algorithms Wright, Stephen J. Mathematical Programming, Vol. 151, Issue 1 https://doi.org/10.1007/s10107-015-0892-3	journal	March 2015
Data Intensive and Network Aware (DIANA) Grid Scheduling McClatchey, Richard; Anjum, Ashiq; Stockinger, Heinz Journal of Grid Computing, Vol. 5, Issue 1 https://doi.org/10.1007/s10723-006-9059-z	journal	January 2007
SCIP: solving constraint integer programs Achterberg, Tobias Mathematical Programming Computation, Vol. 1, Issue 1 https://doi.org/10.1007/s12532-008-0001-1	journal	January 2009
Data management and transfer in high-performance computational grid environments Allcock, Bill; Bester, Joe; Bresnahan, John Parallel Computing, Vol. 28, Issue 5 https://doi.org/10.1016/S0167-8191(02)00094-7	journal	May 2002
A Bee Colony based optimization approach for simultaneous job scheduling and data replication in grid environments Taheri, Javid; Choon Lee, Young; Zomaya, Albert Y. Computers & Operations Research, Vol. 40, Issue 6 https://doi.org/10.1016/j.cor.2011.11.012	journal	June 2013
The impact of data replication on job scheduling performance in the Data Grid Tang, Ming; Lee, Bu-Sung; Tang, Xueyan Future Generation Computer Systems, Vol. 22, Issue 3 https://doi.org/10.1016/j.future.2005.08.004	journal	February 2006
Job scheduling and data replication on data grids Chang, Ruay-Shiung; Chang, Jih-Sheng; Lin, Shin-Yi Future Generation Computer Systems, Vol. 23, Issue 7 https://doi.org/10.1016/j.future.2007.02.008	journal	August 2007
Computational models and heuristic methods for Grid scheduling problems Xhafa, Fatos; Abraham, Ajith Future Generation Computer Systems, Vol. 26, Issue 4 https://doi.org/10.1016/j.future.2009.11.005	journal	April 2010
A survey of dynamic replication strategies for improving data availability in data grids Amjad, Tehmina; Sher, Muhammad; Daud, Ali Future Generation Computer Systems, Vol. 28, Issue 2 https://doi.org/10.1016/j.future.2011.06.009	journal	February 2012
Hopfield neural network for simultaneous job scheduling and data replication in grids Taheri, Javid; Zomaya, Albert Y.; Bouvry, Pascal Future Generation Computer Systems, Vol. 29, Issue 8 https://doi.org/10.1016/j.future.2013.04.020	journal	October 2013
A balanced scheduler with data reuse and replication for scientific workflows in cloud computing systems Casas, Israel; Taheri, Javid; Ranjan, Rajiv Future Generation Computer Systems, Vol. 74 https://doi.org/10.1016/j.future.2015.12.005	journal	September 2017
Joint optimization of multi-dimensional resource allocation and task offloading for QoE enhancement in Cloud-Edge-End collaboration Zeng, Chao; Wang, Xingwei; Zeng, Rongfei Future Generation Computer Systems, Vol. 155 https://doi.org/10.1016/j.future.2024.01.025	journal	June 2024
Swarm scheduling approaches for work-flow applications with security constraints in distributed data-intensive computing environments Liu, Hongbo; Abraham, Ajith; Snášel, Václav Information Sciences, Vol. 192 https://doi.org/10.1016/j.ins.2011.12.032	journal	June 2012
Heuristics for periodical batch job scheduling in a MapReduce computing framework Li, Xiaoping; Jiang, Tianze; Ruiz, Rubén Information Sciences, Vol. 326 https://doi.org/10.1016/j.ins.2015.07.040	journal	January 2016
A survey on resource allocation in high performance distributed computing systems Hussain, Hameed; Malik, Saif Ur Rehman; Hameed, Abdul Parallel Computing, Vol. 39, Issue 11 https://doi.org/10.1016/j.parco.2013.09.009	journal	November 2013
Grid Computing for Earth Science Renard, Philippe; Badoux, Vincent; Petitdidier, Monique Eos, Transactions American Geophysical Union, Vol. 90, Issue 14 https://doi.org/10.1029/2009EO140002	journal	April 2009
Power laws, Pareto distributions and Zipf's law Newman, Mej Contemporary Physics, Vol. 46, Issue 5 https://doi.org/10.1080/00107510500052444	journal	September 2005
Grid Computing for Bioinformatics and Computational Biology. Wiley Series on Bioinformatics: Computational Techniques and Engineering. Edited by El-Ghazali Talbi and , Albert Y. Zomaya. Hoboken (New Jersey): John Wiley and Sons. $125.00. xv + 392 p.; ill.; index. 978‐0‐471‐78409‐8. 2008. Au, Loretta The Quarterly Review of Biology, Vol. 84, Issue 1 https://doi.org/10.1086/598260	journal	March 2009
Adaptive Replica Placement in Hierarchical Data Grids Shorfuzzaman, Mohammad; Graham, Peter; Eskicioglu, Rasit Journal of Physics: Conference Series, Vol. 256 https://doi.org/10.1088/1742-6596/256/1/012020	journal	November 2010
Overview of ATLAS PanDA Workload Management Maeno, T.; De, K.; Wenaus, T. Journal of Physics: Conference Series, Vol. 331, Issue 7 https://doi.org/10.1088/1742-6596/331/7/072024	journal	December 2011
A local update strategy for iterative reconstruction from projections Sauer, K.; Bouman, C. IEEE Transactions on Signal Processing, Vol. 41, Issue 2 https://doi.org/10.1109/78.193196	journal	January 1993
Data-Aware Task Allocation for Achieving Low Latency in Collaborative Edge Computing Sahni, Yuvraj; Cao, Jiannong; Yang, Lei IEEE Internet of Things Journal, Vol. 6, Issue 2 https://doi.org/10.1109/JIOT.2018.2886757	journal	April 2019
AI Surrogate Model for Distributed Computing Workloads Park, David K.; Ren, Yihui; Kilic, Ozgur O. SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SCW63240.2024.00018	conference	November 2024
Joint Optimization Across Timescales: Resource Placement and Task Dispatching in Edge Clouds Wei, Xinliang; Rahman, A. B. M. Mohaimenur; Cheng, Dazhao IEEE Transactions on Cloud Computing, Vol. 11, Issue 1 https://doi.org/10.1109/TCC.2021.3113605	journal	January 2023
Incorporating Intelligence in Fog Computing for Big Data Analysis in Smart Cities Tang, Bo; Chen, Zhen; Hefferman, Gerald IEEE Transactions on Industrial Informatics, Vol. 13, Issue 5 https://doi.org/10.1109/TII.2017.2679740	journal	October 2017
Cyclic coordinate descent: A robotics algorithm for protein loop closure Canutescu, Adrian A.; Dunbrack, Roland L. Protein Science, Vol. 12, Issue 5 https://doi.org/10.1110/ps.0242703	journal	May 2003
Efficiency of Coordinate Descent Methods on Huge-Scale Optimization Problems Nesterov, Yu. SIAM Journal on Optimization, Vol. 22, Issue 2 https://doi.org/10.1137/100802001	journal	January 2012
A survey of Web cache replacement strategies Podlipnig, Stefan; Böszörmenyi, Laszlo ACM Computing Surveys, Vol. 35, Issue 4 https://doi.org/10.1145/954339.954341	journal	December 2003
A Multiplier Adjustment Method for the Generalized Assignment Problem Fisher, Marshall L.; Jaikumar, R.; Van Wassenhove, Luk N. Management Science, Vol. 32, Issue 9 https://doi.org/10.1287/mnsc.32.9.1095	journal	September 1986
Mixed Integer Quadratic Programming Based Scheduling Methods for Day-Ahead Bidding and Intra-Day Operation of Virtual Power Plant Ko, Rakkyung; Kang, Daeyoung; Joo, Sung-Kwan Energies, Vol. 12, Issue 8 https://doi.org/10.3390/en12081410	journal	April 2019
A Survey on Scheduling Heuristics in Grid Computing Environment No authors listed International Journal of Modern Education and Computer Science, Vol. 6, Issue 10 https://doi.org/10.5815/ijmecs.2014.10.08	journal	October 2014

Similar Records

PMU and Communication Infrastructure Restoration for Post-Attack Observability Recovery of Power Grids

Conference · Tue Dec 31 23:00:00 EST 2019 · OSTI ID:1871640

Solution of Mixed-Integer Programming Problems on the XT5

Conference · Wed Dec 31 23:00:00 EST 2008 · OSTI ID:979207

Related Subjects

97 MATHEMATICS AND COMPUTING
Date allocation
Grid computing environments
High performance computing
Job scheduling
Mixed integer linear programming

Alternative mixed integer linear programming optimization for joint job scheduling and data allocation in grid computing

Citation Formats

References (34)

Similar Records

Related Subjects