PUMIPic: A mesh-based approach to unstructured mesh Particle-In-Cell on GPUs

Diamond, Gerrett; Smith, Cameron W.; Zhang, Chonglin; Yoon, Eisung; Shephard, Mark S.

doi:10.1016/j.jpdc.2021.06.004

PUMIPic: A mesh-based approach to unstructured mesh Particle-In-Cell on GPUs

Journal Article · Wed Jun 16 00:00:00 EDT 2021 · Journal of Parallel and Distributed Computing

DOI:https://doi.org/10.1016/j.jpdc.2021.06.004· OSTI ID:1977384

^[1]; Smith, Cameron W. ^[2]; Zhang, Chonglin ^[2]; Yoon, Eisung ^[3]; Shephard, Mark S. ^[2]

Rensselaer Polytechnic Inst., Troy, NY (United States); OSTI
Rensselaer Polytechnic Inst., Troy, NY (United States)
Ulsan National Institute of Science and Technology (UNIST), Ulsan (Korea, Republic of)

Unstructured mesh particle-in-cell, PIC, simulations executing on the current and next generation of massively parallel systems require new methods for both the mesh and particles to achieve performance and scalability on GPUs. The traditional approach to implementing PIC simulations defines data structures and algorithms in terms of particles with a full copy of the unstructured mesh on every process. To effectively scale the unstructured mesh and particles, mesh-based PIC uses the unstructured mesh as the predominant data structure with the particles stored in terms of the mesh entities. Here, this paper details the PUMIPic library, a framework for developing efficient and performance-portable mesh-based PIC simulations on GPU systems. A pseudo physics simulation based on a five-dimensional gyro-kinetic code for modeling plasma physics is used to examine the performance of PUMIPic. Scaling studies of the unstructured mesh partition and number of particles are performed up to 4096 nodes of the Summit system at Oak Ridge National Laboratory. The studies show that mesh-based PIC can utilize a partitioned mesh and maintain scaling up to system limitations.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Rensselaer Polytechnic Inst., Troy, NY (United States); UT-Battelle LLC/ORNL, Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC05-00OR22725; AC52-07NA27344; SC0018275

OSTI ID:: 1977384

Journal Information:: Journal of Parallel and Distributed Computing, Journal Name: Journal of Parallel and Distributed Computing Journal Issue: C Vol. 157; ISSN 0743-7315

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (26)

A General Topology-Based mesh data Structure Beall, Mark W.; Shephard, Mark S. International Journal for Numerical Methods in Engineering, Vol. 40, Issue 9 https://doi.org/10.1002/(SICI)1097-0207(19970515)40:9<1573::AID-NME128>3.0.CO;2-9	journal	May 1997
Particle tracking in unstructured, arbitrary polyhedral meshes for use in CFD and molecular dynamics Macpherson, Graham B.; Nordin, Niklas; Weller, Henry G. Communications in Numerical Methods in Engineering, Vol. 25, Issue 3 https://doi.org/10.1002/cnm.1128	journal	March 2009
A new approach for sparse matrix vector product on NVIDIA GPUs Vázquez, F.; Fernández, J. J.; Garzón, E. M. Concurrency and Computation: Practice and Experience, Vol. 23, Issue 8 https://doi.org/10.1002/cpe.1658	journal	September 2010
Performance of dynamic load balancing algorithms for unstructured mesh calculations Williams, Roy D. Concurrency: Practice and Experience, Vol. 3, Issue 5 https://doi.org/10.1002/cpe.4330030502	journal	October 1991
Gyrokinetic particle simulation model Lee, W. W. Journal of Computational Physics, Vol. 72, Issue 1 https://doi.org/10.1016/0021-9991(87)90080-5	journal	September 1987
Dynamic load balancing for distributed memory multiprocessors Cybenko, George Journal of Parallel and Distributed Computing, Vol. 7, Issue 2 https://doi.org/10.1016/0743-7315(89)90021-X	journal	October 1989
An efficient particle-locating algorithm for application in arbitrary 2D and 3D grids Chordá, R.; Blasco, J. A.; Fueyo, N. International Journal of Multiphase Flow, Vol. 28, Issue 9 https://doi.org/10.1016/S0301-9322(02)00045-9	journal	September 2002
Particle-field decomposition and domain decomposition in parallel particle-in-cell beam dynamics simulation Qiang, Ji; Li, Xiaoye Computer Physics Communications, Vol. 181, Issue 12 https://doi.org/10.1016/j.cpc.2010.08.021	journal	December 2010
hPIC: A scalable electrostatic Particle-in-Cell for Plasma–Material Interactions Khaziev, Rinat; Curreli, Davide Computer Physics Communications, Vol. 229 https://doi.org/10.1016/j.cpc.2018.03.028	journal	August 2018
GITR: An accelerated global scale particle tracking code for wall material erosion and redistribution in fusion relevant plasma–material interactions Younkin, T. R.; Green, D. L.; Simpson, A. B. Computer Physics Communications, Vol. 264 https://doi.org/10.1016/j.cpc.2021.107885	journal	July 2021
A new hybrid-Lagrangian numerical scheme for gyrokinetic simulation of tokamak edge plasma Ku, S.; Hager, R.; Chang, C. S. Journal of Computational Physics, Vol. 315 https://doi.org/10.1016/j.jcp.2016.03.062	journal	June 2016
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Full-f XGC1 gyrokinetic study of improved ion energy confinement from impurity stabilization of ITG turbulence Kim, Kyuho; Kwon, Jae-Min; Chang, C. S. Physics of Plasmas, Vol. 24, Issue 6 https://doi.org/10.1063/1.4984991	journal	June 2017
A fast low-to-high confinement mode bifurcation dynamics in the boundary-plasma gyrokinetic code XGC1 Ku, S.; Chang, C. S.; Hager, R. Physics of Plasmas, Vol. 25, Issue 5 https://doi.org/10.1063/1.5020792	journal	May 2018
Gyroaveraging operations using adaptive matrix operators Dominski, Julien; Ku, Seung-Hoe; Chang, Choong-Seock Physics of Plasmas, Vol. 25, Issue 5 https://doi.org/10.1063/1.5026767	journal	May 2018
Cross-verification of the global gyrokinetic codes GENE and XGC Merlo, G.; Dominski, J.; Bhattacharjee, A. Physics of Plasmas, Vol. 25, Issue 6 https://doi.org/10.1063/1.5036563	journal	June 2018
Scaling to 150K cores: Recent algorithm and performance engineering developments enabling XGC1 to run at scale Adams, Mark F.; Ku, Seung-Hoe; Worley, Patrick Journal of Physics: Conference Series, Vol. 180 https://doi.org/10.1088/1742-6596/180/1/012036	journal	July 2009
Method for solving the gyrokinetic Poisson equation in general geometry Lin, Z.; Lee, W. W. Physical Review E, Vol. 52, Issue 5 https://doi.org/10.1103/PhysRevE.52.5646	journal	November 1995
A Partitioning Strategy for Nonuniform Problems on Multiprocessors No authors listed IEEE Transactions on Computers, Vol. C-36, Issue 5 https://doi.org/10.1109/TC.1987.1676942	journal	May 1987
PIConGPU: A Fully Relativistic Particle-in-Cell Code for a GPU Cluster Burau, Heiko; Widera, Renée; Honig, Wolfgang IEEE Transactions on Plasma Science, Vol. 38, Issue 10 https://doi.org/10.1109/TPS.2010.2064310	journal	October 2010
Efficient Parallelization of a Three-Dimensional High-Order Particle-in-Cell Method for the Simulation of a 170 GHz Gyrotron Resonator Neudorfer, Jonathan; Stock, Andreas; Schneider, Rudolf IEEE Transactions on Plasma Science, Vol. 41, Issue 1 https://doi.org/10.1109/TPS.2012.2229298	journal	January 2013
Controlling Unstructured Mesh Partitions for Massively Parallel Simulations Zhou, Min; Sahni, Onkar; Devine, Karen D. SIAM Journal on Scientific Computing, Vol. 32, Issue 6 https://doi.org/10.1137/090777323	journal	January 2010
A Unified Sparse Matrix Data Format for Efficient General Sparse Matrix-Vector Multiplication on Modern Processors with Wide SIMD Units Kreutzer, Moritz; Hager, Georg; Wellein, Gerhard SIAM Journal on Scientific Computing, Vol. 36, Issue 5 https://doi.org/10.1137/130930352	journal	January 2014
Improving Unstructured Mesh Partitions for Multiple Criteria Using Mesh Adjacencies Smith, Cameron W.; Rasquin, Michel; Ibanez, Dan SIAM Journal on Scientific Computing, Vol. 40, Issue 1 https://doi.org/10.1137/15M1027814	journal	January 2018
Parallel Multilevel series k-Way Partitioning Scheme for Irregular Graphs Karypis, George; Kumar, Vipin SIAM Review, Vol. 41, Issue 2 https://doi.org/10.1137/S0036144598334138	journal	January 1999
Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers Wang, Bei; Ethier, Stephane; Tang, William The International Journal of High Performance Computing Applications, Vol. 33, Issue 1 https://doi.org/10.1177/1094342017712059	journal	June 2017

Similar Records

Center for Tokamak Transient Simulations (RPI Unstructured Mesh Developments for FES SciDAC4 Partnerships) (Final Report)

Technical Report · Thu Jun 27 00:00:00 EDT 2024 · OSTI ID:2336685

Improving Unstructured Mesh Partitions for Multiple Criteria Using Mesh Adjacencies

Journal Article · Mon Feb 12 19:00:00 EST 2018 · SIAM Journal on Scientific Computing · OSTI ID:1438033

Geometric electrostatic particle-in-cell algorithm on unstructured meshes

Journal Article · Tue Jul 20 20:00:00 EDT 2021 · Journal of Plasma Physics · OSTI ID:1814591

Related Subjects

97 MATHEMATICS AND COMPUTING
GPU
Particle-in-cell
Unstructured mesh

PUMIPic: A mesh-based approach to unstructured mesh Particle-In-Cell on GPUs

Citation Formats

References (26)

Similar Records

Related Subjects