DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Dynamic load balancing for a mesh‐based scientific application

Abstract

Summary CMT‐nek is a new scientific application for performing high fidelity predictive simulations of particle‐laden, explosively dispersed turbulent flows. CMT‐nek is compute‐intensive and targeted for deployment on exascale platforms. The moving particles are the primary source of load imbalance when the application is executed on parallel processors. In a demonstration problem, all the particles are initially in a closed container until a detonation occurs and the particles move apart. If all processors get an equal share of the fluid domain, then only some of the processors get sections of the domain that are initially laden with particles, leading to disparate loads on the processors. To eliminate load imbalance in different processors and to speed up the makespan, we present different load‐balancing algorithms for CMT‐nek on large‐scale multicore platforms. The load on a processor is determined using different techniques. The performance of the different load‐balancing algorithms is compared, and the associated overheads are analyzed. Evaluations of the application with and without load‐balancing are conducted, and these show that with load‐balancing, simulation time becomes faster by a factor of up to 9.97. The performance was further improved by a factor of up to 1.42 using machine‐learning–based algorithms.

Authors:
ORCiD logo [1];  [1];  [2];  [2];  [2];  [1]
  1. Department of Computer &, Information Science &, Engineering University of Florida Gainesville Florida
  2. Department of Mechanical &, Aerospace Engineering University of Florida Gainesville Florida
Publication Date:
Sponsoring Org.:
USDOE
OSTI Identifier:
1581451
Resource Type:
Publisher's Accepted Manuscript
Journal Name:
Concurrency and Computation. Practice and Experience
Additional Journal Information:
Journal Name: Concurrency and Computation. Practice and Experience Journal Volume: 32 Journal Issue: 9; Journal ID: ISSN 1532-0626
Publisher:
Wiley Blackwell (John Wiley & Sons)
Country of Publication:
United Kingdom
Language:
English

Citation Formats

Zhai, Keke, Banerjee, Tania, Zwick, David, Hackl, Jason, Koneru, Rahul, and Ranka, Sanjay. Dynamic load balancing for a mesh‐based scientific application. United Kingdom: N. p., 2020. Web. doi:10.1002/cpe.5626.
Zhai, Keke, Banerjee, Tania, Zwick, David, Hackl, Jason, Koneru, Rahul, & Ranka, Sanjay. Dynamic load balancing for a mesh‐based scientific application. United Kingdom. https://doi.org/10.1002/cpe.5626
Zhai, Keke, Banerjee, Tania, Zwick, David, Hackl, Jason, Koneru, Rahul, and Ranka, Sanjay. Mon . "Dynamic load balancing for a mesh‐based scientific application". United Kingdom. https://doi.org/10.1002/cpe.5626.
@article{osti_1581451,
title = {Dynamic load balancing for a mesh‐based scientific application},
author = {Zhai, Keke and Banerjee, Tania and Zwick, David and Hackl, Jason and Koneru, Rahul and Ranka, Sanjay},
abstractNote = {Summary CMT‐nek is a new scientific application for performing high fidelity predictive simulations of particle‐laden, explosively dispersed turbulent flows. CMT‐nek is compute‐intensive and targeted for deployment on exascale platforms. The moving particles are the primary source of load imbalance when the application is executed on parallel processors. In a demonstration problem, all the particles are initially in a closed container until a detonation occurs and the particles move apart. If all processors get an equal share of the fluid domain, then only some of the processors get sections of the domain that are initially laden with particles, leading to disparate loads on the processors. To eliminate load imbalance in different processors and to speed up the makespan, we present different load‐balancing algorithms for CMT‐nek on large‐scale multicore platforms. The load on a processor is determined using different techniques. The performance of the different load‐balancing algorithms is compared, and the associated overheads are analyzed. Evaluations of the application with and without load‐balancing are conducted, and these show that with load‐balancing, simulation time becomes faster by a factor of up to 9.97. The performance was further improved by a factor of up to 1.42 using machine‐learning–based algorithms.},
doi = {10.1002/cpe.5626},
journal = {Concurrency and Computation. Practice and Experience},
number = 9,
volume = 32,
place = {United Kingdom},
year = {Mon Jan 06 00:00:00 EST 2020},
month = {Mon Jan 06 00:00:00 EST 2020}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1002/cpe.5626

Citation Metrics:
Cited by: 1 work
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

A load-balancing algorithm for a parallel electromagnetic particle-in-cell code
journal, May 2003


Automated Load Balancing Invocation Based on Application Characteristics
conference, September 2012

  • Menon, Harshitha; Jain, Nikhil; Zheng, Gengbin
  • 2012 IEEE International Conference on Cluster Computing (CLUSTER)
  • DOI: 10.1109/CLUSTER.2012.61

Hierarchical Load Balancing for Charm++ Applications on Large Supercomputers
conference, September 2010

  • Zheng, Gengbin; Meneses, Esteban; Bhatele, Abhinav
  • 2010 International Conference on Parallel Processing Workshops (ICPPW), 2010 39th International Conference on Parallel Processing Workshops
  • DOI: 10.1109/ICPPW.2010.65

Architecture-independent locality-improving transformations of computational graphs embedded in k -dimensions
conference, January 1995

  • Ou, Chao-Wei; Gunwani, Manoj; Ranka, Sanjay
  • Proceedings of the 9th international conference on Supercomputing - ICS '95
  • DOI: 10.1145/224538.224573

Automatic data distribution and load balancing with space-filling curves: implementation in CONQUEST
journal, June 2008


Spectral Methods: Evolution to Complex Geometrics and Applications to Fluid Dynamics
book, January 2007


Load balancing n-body simulations with highly non-uniform density
conference, January 2014

  • Pearce, Olga; Gamblin, Todd; de Supinski, Bronis R.
  • Proceedings of the 28th ACM international conference on Supercomputing - ICS '14
  • DOI: 10.1145/2597652.2597659

GROMACS 4:  Algorithms for Highly Efficient, Load-Balanced, and Scalable Molecular Simulation
journal, February 2008

  • Hess, Berk; Kutzner, Carsten; van der Spoel, David
  • Journal of Chemical Theory and Computation, Vol. 4, Issue 3
  • DOI: 10.1021/ct700301q

Terascale spectral element algorithms and implementations
conference, January 1999

  • Tufo, H. M.; Fischer, P. F.
  • Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '99
  • DOI: 10.1145/331532.331599

POSTER: Automated Load Balancer Selection Based on Application Characteristics
journal, January 2017

  • Menon, Harshitha; Chandrasekar, Kavitha; Kale, Laxmikant V.
  • ACM SIGPLAN Notices, Vol. 52, Issue 8
  • DOI: 10.1145/3155284.3019033

Dynamic Load Balancing for Unstructured Meshes on Space-Filling Curves
conference, May 2012

  • Harlacher, Daniel F.; Klimach, Harald; Roller, Sabine
  • 2012 26th IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum
  • DOI: 10.1109/IPDPSW.2012.207

Honey bee behavior inspired load balancing of tasks in cloud computing environments
journal, May 2013


CMT-Bone — A Proxy Application for Compressible Multiphase Turbulent Flows
conference, December 2016

  • Banerjee, Tania; Hackl, Jason; Shringarpure, Mrugesh
  • 2016 IEEE 23rd International Conference on High Performance Computing (HiPC)
  • DOI: 10.1109/HiPC.2016.029

Dynamic topology aware load balancing algorithms for molecular dynamics applications
conference, January 2009

  • Bhatelé, Abhinav; Kalé, Laxmikant V.; Kumar, Sameer
  • Proceedings of the 23rd international conference on Conference on Supercomputing - ICS '09
  • DOI: 10.1145/1542275.1542295

Dynamic Load Balancing for Compressible Multiphase Turbulence
conference, January 2018

  • Zhai, Keke; Banerjee, Tania; Zwick, David
  • Proceedings of the 2018 International Conference on Supercomputing - ICS '18
  • DOI: 10.1145/3205289.3205304

Quantifying load imbalance on virtualized enterprise servers
conference, January 2010

  • Arzuaga, Emmanuel; Kaeli, David R.
  • Proceedings of the first joint WOSP/SIPEW international conference on Performance engineering - WOSP/SIPEW '10
  • DOI: 10.1145/1712605.1712641

A load index and load balancing algorithm for heterogeneous clusters
journal, February 2013

  • Bosque, Jose Luis; Toharia, Pablo; Robles, Oscar D.
  • The Journal of Supercomputing, Vol. 65, Issue 3
  • DOI: 10.1007/s11227-013-0881-3

Analysis of scalable data-privatization threading algorithms for hybrid MPI/OpenMP parallelization of molecular dynamics
journal, April 2013

  • Kunaseth, Manaschai; Richards, David F.; Glosli, James N.
  • The Journal of Supercomputing, Vol. 66, Issue 1
  • DOI: 10.1007/s11227-013-0915-x

A Dynamic Load Balancing Framework for Real-time Applications in Message Passing Systems
journal, May 2010

  • El Kabbany, Ghada F.; Wanas, Nayer M.; Hegazi, Nadia H.
  • International Journal of Parallel Programming, Vol. 39, Issue 2
  • DOI: 10.1007/s10766-010-0134-5

Software support for irregular and loosely synchronous problems
journal, January 1992


Barycentric Lagrange Interpolation
journal, January 2004


High-Order Methods for Incompressible Fluid Flow
book, January 2009


OhHelp: a scalable domain-decomposing dynamic load balancing for particle-in-cell simulations
conference, January 2009

  • Nakashima, Hiroshi; Miyake, Yohei; Usui, Hideyuki
  • Proceedings of the 23rd international conference on Conference on Supercomputing - ICS '09
  • DOI: 10.1145/1542275.1542293

Total variation diminishing Runge-Kutta schemes
journal, January 1998

  • Gottlieb, Sigal; Shu, Chi-Wang
  • Mathematics of Computation of the American Mathematical Society, Vol. 67, Issue 221
  • DOI: 10.1090/S0025-5718-98-00913-2

Fast Parallel Direct Solvers for Coarse Grid Problems
journal, February 2001

  • Tufo, H. M.; Fischer, P. F.
  • Journal of Parallel and Distributed Computing, Vol. 61, Issue 2
  • DOI: 10.1006/jpdc.2000.1676

Dynamic Load Balancing for a 2D Concurrent Plasma PIC Code
journal, December 1993

  • Ferraro, Robert D.; Liewer, Paulett C.; Decyk, Viktor K.
  • Journal of Computational Physics, Vol. 109, Issue 2
  • DOI: 10.1006/jcph.1993.1221

Highly scalable SFC-based dynamic load balancing and its application to atmospheric modeling
journal, May 2018


Quantifying the effectiveness of load balance algorithms
conference, January 2012

  • Pearce, Olga; Gamblin, Todd; de Supinski, Bronis R.
  • Proceedings of the 26th ACM international conference on Supercomputing - ICS '12
  • DOI: 10.1145/2304576.2304601

A genetic algorithm based autotuning approach for performance and energy optimization
conference, December 2015

  • Banerjee, Tania; Ranka, Sanjay
  • 2015 Sixth International Green and Sustainable Computing Conference (IGSC)
  • DOI: 10.1109/IGCC.2015.7393708

Towards energy-aware scheduling in data centers using machine learning
conference, January 2010

  • Berral, Josep Ll.; Goiri, Íñigo; Nou, Ramón
  • Proceedings of the 1st International Conference on Energy-Efficient Computing and Networking - e-Energy '10
  • DOI: 10.1145/1791314.1791349

Parallel Remapping of Adaptive Problems
journal, May 1997

  • Ou, Chao-Wei; Ranka, Sanjay
  • Journal of Parallel and Distributed Computing, Vol. 42, Issue 2
  • DOI: 10.1006/jpdc.1997.1314

Fast and parallel mapping algorithms for irregular problems
journal, January 1996

  • Ou, Chao-Wei; Ranka, Sanjay; Fox, Geoffrey
  • The Journal of Supercomputing, Vol. 10, Issue 2
  • DOI: 10.1007/BF00130706

The Plasma Simulation Code: A modern particle-in-cell code with patch-based load-balancing
journal, August 2016

  • Germaschewski, Kai; Fox, William; Abbott, Stephen
  • Journal of Computational Physics, Vol. 318
  • DOI: 10.1016/j.jcp.2016.05.013

Parallel construction of multidimensional binary search trees
journal, January 2000

  • Al-Furajh, I.; Aluru, S.; Goil, S.
  • IEEE Transactions on Parallel and Distributed Systems, Vol. 11, Issue 2
  • DOI: 10.1109/71.841750

Energy efficiency of load balancing for data-parallel applications in heterogeneous systems
journal, September 2016

  • Pérez, Borja; Stafford, Esteban; Bosque, José Luis
  • The Journal of Supercomputing, Vol. 73, Issue 1
  • DOI: 10.1007/s11227-016-1864-y