Dynamic load balancing with enhanced shared-memory parallelism for particle-in-cell codes

Miller, Kyle G.; Lee, Roman P.; Tableman, Adam; Helm, Anton; Fonseca, Ricardo A.; Decyk, Viktor K.; Mori, Warren B.

doi:10.1016/j.cpc.2020.107633

Title: Dynamic load balancing with enhanced shared-memory parallelism for particle-in-cell codes

Journal Article · Fri Sep 25 00:00:00 EDT 2020 · Computer Physics Communications

DOI:https://doi.org/10.1016/j.cpc.2020.107633· OSTI ID:1784762

^[1];

^[1]; Tableman, Adam ^[1];

^[2];

^[3]; Decyk, Viktor K. ^[1]; Mori, Warren B. ^[1]

Univ. of California, Los Angeles, CA (United States)
Inst. Superior Técnico, Lisboa (Portugal)
ISCTE - Inst. Univ. de Lisboa (Portugal); Inst. Superior Técnico, Lisboa (Portugal)

Furthering our understanding of many of today’s interesting problems in plasma physics – including plasma based acceleration and magnetic reconnection with pair production due to quantum electrodynamic effects – requires large-scale kinetic simulations using particle-in-cell (PIC) codes. However, these simulations are extremely demanding, requiring that contemporary PIC codes be designed to efficiently use a new fleet of exascale computing architectures. To this end, the key issue of parallel load balance across computational nodes must be addressed. We discuss the implementation of dynamic load balancing by dividing the simulation space into many small, self-contained regions or ‘‘tiles,’’ along with shared-memory (e.g., OpenMP) parallelism both over many tiles and within single tiles. The load balancing algorithm can be used with three different topologies, including two space-filling curves. Here, we tested this implementation in the code Osiris and show low overhead and improved scalability with OpenMP thread number on simulations with both uniform load and severe load imbalance. Compared to other load-balancing techniques, our algorithm gives order-of-magnitude improvement in parallel scalability for simulations with severe load imbalance issues.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Univ. of California, San Diego, La Jolla, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF); Fundação para a Ciência e Tecnologia; European Research Council (ERC)

Grant/Contract Number:: NA0003842; ACI-1339893; 1806046; SC0019010; SC0010064; B634451; B635445; PTDC-FIS-PLA-2940-2014; 695008

OSTI ID:: 1784762

Alternate ID(s):: OSTI ID: 1775677

Journal Information:: Computer Physics Communications, Vol. 259; ISSN 0010-4655

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (26)

Laser Electron Accelerator Tajima, T.; Dawson, J. M. Physical Review Letters, Vol. 43, Issue 4 https://doi.org/10.1103/PhysRevLett.43.267	journal	July 1979
Generating high quality ultrarelativistic electron beams using an evolving electron beam driver Dalichaouch, T. N.; Xu, X. L.; Li, F. Physical Review Accelerators and Beams, Vol. 23, Issue 2 https://doi.org/10.1103/PhysRevAccelBeams.23.021304	journal	February 2020
Ueber die stetige Abbildung einer Line auf ein Fl�chenst�ck Hilbert, David Mathematische Annalen, Vol. 38, Issue 3 https://doi.org/10.1007/BF01199431	journal	September 1891
Load-balancing spatially located computations using rectangular partitions Saule, Erik; Baş, Erdeniz Ö.; Çatalyürek, Ümit V. Journal of Parallel and Distributed Computing, Vol. 72, Issue 10 https://doi.org/10.1016/j.jpdc.2012.05.013	journal	October 2012
High quality electron bunch generation using a longitudinal density-tailored plasma-based accelerator in the three-dimensional blowout regime Xu, X. L.; Li, F.; An, W. Physical Review Accelerators and Beams, Vol. 20, Issue 11 https://doi.org/10.1103/PhysRevAccelBeams.20.111303	journal	November 2017
Particle-in-Cell algorithms for emerging computer architectures Decyk, Viktor K.; Singh, Tajendra V. Computer Physics Communications, Vol. 185, Issue 3 https://doi.org/10.1016/j.cpc.2013.10.013	journal	March 2014
Smilei : A collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation Derouillat, J.; Beck, A.; Pérez, F. Computer Physics Communications, Vol. 222 https://doi.org/10.1016/j.cpc.2017.09.024	journal	January 2018
Development of the indirect‐drive approach to inertial confinement fusion and the target physics basis for ignition and gain Lindl, John Physics of Plasmas, Vol. 2, Issue 11 https://doi.org/10.1063/1.871025	journal	November 1995
Dynamic Load Balancing for a 2D Concurrent Plasma PIC Code Ferraro, Robert D.; Liewer, Paulett C.; Decyk, Viktor K. Journal of Computational Physics, Vol. 109, Issue 2 https://doi.org/10.1006/jcph.1993.1221	journal	December 1993
Ionization-Induced Electron Trapping in Ultrarelativistic Plasma Wakes Oz, E.; Deng, S.; Katsouleas, T. Physical Review Letters, Vol. 98, Issue 8 https://doi.org/10.1103/PhysRevLett.98.084801	journal	February 2007
Heating uniformity of a microwave discharge plasma to redistribute a solid fuel layer inside a cryogenic target for inertial confinement fusion Chen, C. M.; Norimatsu, T.; Izawa, Y. Journal of Vacuum Science & Technology A: Vacuum, Surfaces, and Films, Vol. 13, Issue 6 https://doi.org/10.1116/1.579612	journal	November 1995
A general concurrent algorithm for plasma particle-in-cell simulation codes Liewer, Paulett C.; Decyk, Viktor K. Journal of Computational Physics, Vol. 85, Issue 2 https://doi.org/10.1016/0021-9991(89)90153-8	journal	December 1989
Inertial-confinement fusion with lasers Betti, R.; Hurricane, O. A. Nature Physics, Vol. 12, Issue 5 https://doi.org/10.1038/nphys3736	journal	May 2016
The Plasma Simulation Code: A modern particle-in-cell code with patch-based load-balancing Germaschewski, Kai; Fox, William; Abbott, Stephen Journal of Computational Physics, Vol. 318 https://doi.org/10.1016/j.jcp.2016.05.013	journal	August 2016
Particle-in-cell simulations with charge-conserving current deposition on graphic processing units Kong, Xianglong; Huang, Michael C.; Ren, Chuang Journal of Computational Physics, Vol. 230, Issue 4 https://doi.org/10.1016/j.jcp.2010.11.032	journal	February 2011
Plasma Electron Trapping and Acceleration in a Plasma Wake Field Using a Density Transition Suk, H.; Barov, N.; Rosenzweig, J. B. Physical Review Letters, Vol. 86, Issue 6 https://doi.org/10.1103/PhysRevLett.86.1011	journal	February 2001
Skeleton Particle-in-Cell Codes on Emerging Computer Architectures Decyk, Viktor K. Computing in Science & Engineering, Vol. 17, Issue 2 https://doi.org/10.1109/MCSE.2014.131	journal	March 2015
Generating multi-GeV electron bunches using single stage laser wakefield acceleration in a 3D nonlinear regime Lu, W.; Tzoufras, M.; Joshi, C. Physical Review Special Topics - Accelerators and Beams, Vol. 10, Issue 6 https://doi.org/10.1103/PhysRevSTAB.10.061301	journal	June 2007
Implementation of a hybrid particle code with a PIC description in r–z and a gridless description in ϕ into OSIRIS Davidson, A.; Tableman, A.; An, W. Journal of Computational Physics, Vol. 281 https://doi.org/10.1016/j.jcp.2014.10.064	journal	January 2015
Exploiting multi-scale parallelism for large scale numerical modelling of laser wakefield accelerators Fonseca, R. A.; Vieira, J.; Fiuza, F. Plasma Physics and Controlled Fusion, Vol. 55, Issue 12 https://doi.org/10.1088/0741-3335/55/12/124011	journal	November 2013
Plasma wakefield acceleration experiments at FACET II Joshi, C.; Adli, E.; An, W. Plasma Physics and Controlled Fusion, Vol. 60, Issue 3 https://doi.org/10.1088/1361-6587/aaa2e3	journal	January 2018
Ultrahigh performance three-dimensional electromagnetic relativistic kinetic plasma simulation Bowers, K. J.; Albright, B. J.; Yin, L. Physics of Plasmas, Vol. 15, Issue 5 https://doi.org/10.1063/1.2840133	journal	May 2008
Dynamic load balancing of plasma particle-in-cell simulations: The taskfarm alternative Othmer, C.; Schüle, J. Computer Physics Communications, Vol. 147, Issue 1-2 https://doi.org/10.1016/s0010-4655(02)00389-2	journal	August 2002
Acceleration of Electrons by the Interaction of a Bunched Electron Beam with a Plasma Chen, Pisin; Dawson, J. M.; Huff, Robert W. Physical Review Letters, Vol. 54, Issue 7 https://doi.org/10.1103/physrevlett.54.693	journal	February 1985
Plasma Electron Trapping and Acceleration in a Plasma wake Field Using a Density Transition Suk, H.; Barov, N.; Rosenzweig, J. B. Proceedings of the 2nd ICFA Advanced Accelerator Workshop, The Physics of High Brightness Beams https://doi.org/10.1142/9789812792181_0026	conference	November 2011
SMILEI: a collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation Derouillat, J.; Beck, A.; Pérez, F. arXiv https://doi.org/10.48550/arxiv.1702.05128	text	January 2017

Similar Records

On numerical errors to the fields surrounding a relativistically moving particle in PIC codes

Journal Article · Fri Apr 03 00:00:00 EDT 2020 · Journal of Computational Physics · OSTI ID:1784762

Xu, Xinlu; Li, Fei; Tsung, Frank S.; +7 more

Accurately simulating nine-dimensional phase space of relativistic particles in strong fields

Journal Article · Wed Apr 21 00:00:00 EDT 2021 · Journal of Computational Physics · OSTI ID:1784762

Li, Fei; Decyk, Viktor K.; Miller, Kyle G.; +5 more

A new field solver for modeling of relativistic particle-laser interactions using the particle-in-cell algorithm

Journal Article · Sat Aug 29 00:00:00 EDT 2020 · Computer Physics Communications · OSTI ID:1784762

Li, Fei; Miller, Kyle G.; Xu, Xinlu; +5 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Particle-in-cell (PIC)
Dynamic load balancing
OpenMP parallelization
Plasma kinetic simulation
High-performance computing

Title: Dynamic load balancing with enhanced shared-memory parallelism for particle-in-cell codes

Citation Formats

References (26)

Similar Records

Related Subjects