Acceleration of the particle-in-cell code Osiris with graphics processing units

Lee, Roman P.; Pierce, Jacob R.; Miller, Kyle G.; Almanza, Maria; Tableman, Adam; Decyk, Viktor K.; Fonseca, Ricardo A.; Alves, E. Paulo; Mori, Warren B.

doi:10.1017/s0022377824001569

Title: Acceleration of the particle-in-cell code Osiris with graphics processing units

Journal Article · 07 January 2025 · Journal of Plasma Physics

DOI: https://doi.org/10.1017/s0022377824001569 · OSTI ID:2496787

^[1];

^[2];

^[1]; Tableman, Adam ^[1]; Decyk, Viktor K. ^[1];

^[3]; Alves, E. Paulo ^[1]; Mori, Warren B. ^[1]

University of California, Los Angeles, CA (United States)
University of Rochester, NY (United States)
Instituto Superior Técnico, Lisboa (Portugal); ISCTE – Instituto Universitário de Lisboa (Portugal)

Fully relativistic particle-in-cell (PIC) simulations are crucial for advancing our knowledge of plasma physics. Modern supercomputers based on graphics processing units (GPUs) offer the potential to perform PIC simulations of unprecedented scale, but require robust and feature-rich codes that can fully leverage their computational resources. In this work, this demand is addressed by adding GPU acceleration to the PIC code Osiris. An overview of the algorithm, which features a CUDA extension to the underlying Fortran architecture, is given. Detailed performance benchmarks for thermal plasmas are presented, which demonstrate excellent weak scaling on NERSC's Perlmutter supercomputer and high levels of absolute performance. The robustness of the code to model a variety of physical systems is demonstrated via simulations of Weibel filamentation and laser-wakefield acceleration run with dynamic load balancing. Finally, measurements and analysis of energy consumption are provided that indicate that the GPU algorithm is up to ~14 times faster and ~7 times more energy efficient than the optimized CPU algorithm on a node-to-node basis. The described development addresses the PIC simulation community's computational demands both by contributing a robust and performant GPU-accelerated PIC code and by providing insight into efficient use of GPU hardware.

View Journal Article

Cite

Export

Save

Research Organization:: University of California, San Diego, CA (United States); University of Rochester, NY (United States)

Sponsoring Organization:: European Research Council; Fundação para a Ciência e Tecnologia; US National Science Foundation (NSF); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-05CH11231; NA0003842; NA0004131; NA0004144; NA0004147

OSTI ID:: 2496787

Journal Information:: Journal of Plasma Physics, Journal Name: Journal of Plasma Physics Journal Issue: 1 Vol. 91; ISSN 0022-3778

Publisher:: Cambridge University PressCopyright Statement

Country of Publication:: United States

Language:: English

References (47)

Weighted Particles in Coulomb Collision Simulations Based on the Theory of a Cumulative Scattering Angle Nanbu, K.; Yonemura, S. Journal of Computational Physics, Vol. 145, Issue 2 https://doi.org/10.1006/jcph.1998.6049	journal	September 1998
OSIRIS: A Three-Dimensional, Fully Relativistic Particle in Cell Code for Modeling Plasma Based Accelerators Fonseca, R. A.; Silva, L. O.; Tsung, F. S. Computational Science — ICCS 2002, p. 342-351 https://doi.org/10.1007/3-540-47789-6_36	book	January 2002
Optimize Memory Usage in Vector Particle-In-Cell (VPIC) to Break the 10 Trillion Particle Barrier in Plasma Simulations Tan, Nigel; Bird, Robert; Chen, Guangye Computational Science – ICCS 2021 https://doi.org/10.1007/978-3-030-77964-1_35	book	January 2021
A Bucket Sort Algorithm for the Particle-In-Cell Method on Manycore Architectures Jocksch, Andreas; Hariri, Farah; Tran, Trach-Minh Parallel Processing and Applied Mathematics https://doi.org/10.1007/978-3-319-32149-3_5	book	January 2016
Performance-Portable Many-Core Plasma Simulations: Porting PIConGPU to OpenPower and Beyond Zenker, Erik; Widera, René; Huebl, Axel High Performance Computing https://doi.org/10.1007/978-3-319-46079-6_21	book	October 2016
PIC methods in astrophysics: simulations of relativistic jets and kinetic physics in astrophysical systems Nishikawa, Kenichi; Duţan, Ioana; Köhn, Christoph Living Reviews in Computational Astrophysics, Vol. 7, Issue 1 https://doi.org/10.1007/s41115-021-00012-0	journal	July 2021
A binary collision model for plasma simulation with a particle code Takizuka, Tomonor; Abe, Hirotada Journal of Computational Physics, Vol. 25, Issue 3 https://doi.org/10.1016/0021-9991(77)90099-7	journal	November 1977
Adaptable Particle-in-Cell algorithms for graphical processing units Decyk, Viktor K.; Singh, Tajendra V. Computer Physics Communications, Vol. 182, Issue 3 https://doi.org/10.1016/j.cpc.2010.11.009	journal	March 2011
Fine-sorting one-dimensional particle-in-cell algorithm with Monte-Carlo collisions on a graphics processing unit Mertmann, Philipp; Eremin, Denis; Mussenbrock, Thomas Computer Physics Communications, Vol. 182, Issue 10 https://doi.org/10.1016/j.cpc.2011.05.012	journal	October 2011
Particle-in-Cell algorithms for emerging computer architectures Decyk, Viktor K.; Singh, Tajendra V. Computer Physics Communications, Vol. 185, Issue 3 https://doi.org/10.1016/j.cpc.2013.10.013	journal	March 2014
Particle merging algorithm for PIC codes Vranic, M.; Grismayer, T.; Martins, J. L. Computer Physics Communications, Vol. 191 https://doi.org/10.1016/j.cpc.2015.01.020	journal	June 2015
A portable platform for accelerated PIC codes and its application to GPUs using OpenACC Hariri, F.; Tran, T. M.; Jocksch, A. Computer Physics Communications, Vol. 207 https://doi.org/10.1016/j.cpc.2016.05.008	journal	October 2016
Controlling the numerical Cerenkov instability in PIC simulations using a customized finite difference Maxwell solver and a local FFT based current correction Li, Fei; Yu, Peicheng; Xu, Xinlu Computer Physics Communications, Vol. 214 https://doi.org/10.1016/j.cpc.2017.01.001	journal	May 2017
Smilei : A collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation Derouillat, J.; Beck, A.; Pérez, F. Computer Physics Communications, Vol. 222 https://doi.org/10.1016/j.cpc.2017.09.024	journal	January 2018
A new field solver for modeling of relativistic particle-laser interactions using the particle-in-cell algorithm Li, Fei; Miller, Kyle G.; Xu, Xinlu Computer Physics Communications, Vol. 258 https://doi.org/10.1016/j.cpc.2020.107580	journal	January 2021
Dynamic load balancing with enhanced shared-memory parallelism for particle-in-cell codes Miller, Kyle G.; Lee, Roman P.; Tableman, Adam Computer Physics Communications, Vol. 259 https://doi.org/10.1016/j.cpc.2020.107633	journal	February 2021
HiPACE++: A portable, 3D quasi-static particle-in-cell code Diederichs, S.; Benedetti, C.; Huebl, A. Computer Physics Communications, Vol. 278 https://doi.org/10.1016/j.cpc.2022.108421	journal	September 2022
RaDiO: An efficient spatiotemporal radiation diagnostic for particle-in-cell codes Pardal, M.; Sainte-Marie, A.; Reboul-Salze, A. Computer Physics Communications, Vol. 285 https://doi.org/10.1016/j.cpc.2022.108634	journal	April 2023
Particle-in-cell simulations with charge-conserving current deposition on graphic processing units Kong, Xianglong; Huang, Michael C.; Ren, Chuang Journal of Computational Physics, Vol. 230, Issue 4 https://doi.org/10.1016/j.jcp.2010.11.032	journal	February 2011
An efficient mixed-precision, hybrid CPU–GPU implementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm Chen, G.; Chacón, L.; Barnes, D. C. Journal of Computational Physics, Vol. 231, Issue 16 https://doi.org/10.1016/j.jcp.2012.04.040	journal	June 2012
Implementation of a hybrid particle code with a PIC description in r–z and a gridless description in ϕ into OSIRIS Davidson, A.; Tableman, A.; An, W. Journal of Computational Physics, Vol. 281 https://doi.org/10.1016/j.jcp.2014.10.064	journal	January 2015
On numerical errors to the fields surrounding a relativistically moving particle in PIC codes Xu, Xinlu; Li, Fei; Tsung, Frank S. Journal of Computational Physics, Vol. 413 https://doi.org/10.1016/j.jcp.2020.109451	journal	July 2020
Accurately simulating nine-dimensional phase space of relativistic particles in strong fields Li, Fei; Decyk, Viktor K.; Miller, Kyle G. Journal of Computational Physics, Vol. 438 https://doi.org/10.1016/j.jcp.2021.110367	journal	August 2021
Particle-in-cell plasma simulation on heterogeneous cluster systems Bastrakov, S.; Donchenko, R.; Gonoskov, A. Journal of Computational Science, Vol. 3, Issue 6 https://doi.org/10.1016/j.jocs.2012.08.012	journal	November 2012
Fast parallel Particle-To-Grid interpolation for plasma PIC simulations on the GPU Stantchev, George; Dorland, William; Gumerov, Nail Journal of Parallel and Distributed Computing, Vol. 68, Issue 10 https://doi.org/10.1016/j.jpdc.2008.05.009	journal	October 2008
Graphics processing unit (GPU) programming strategies and trends in GPU computing Brodtkorb, André R.; Hagen, Trond R.; Sætra, Martin L. Journal of Parallel and Distributed Computing, Vol. 73, Issue 1 https://doi.org/10.1016/j.jpdc.2012.04.003	journal	January 2013
Porting WarpX to GPU-accelerated platforms Myers, A.; Almgren, A.; Amorim, L. D. Parallel Computing, Vol. 108 https://doi.org/10.1016/j.parco.2021.102833	journal	December 2021
Modeling of Ionization Physics with the PIC Code OSIRIS Deng, S. AIP Conference Proceedings https://doi.org/10.1063/1.1524875	conference	January 2002
Three-dimensional Weibel instability in astrophysical scenarios Fonseca, Ricardo A.; Silva, Luı́s O.; Tonge, John W. Physics of Plasmas, Vol. 10, Issue 5 https://doi.org/10.1063/1.1556605	journal	May 2003
Towards robust algorithms for current deposition and dynamic load-balancing in a GPU particle in cell code Rossi, Francesco; Londrillo, Pasquale; Sgattoni, Andrea ADVANCED ACCELERATOR CONCEPTS: 15th Advanced Accelerator Concepts Workshop, AIP Conference Proceedings https://doi.org/10.1063/1.4773692	conference	January 2013
Modeling of a chain of three plasma accelerator stages with the WarpX electromagnetic PIC code on GPUs Vay, J. -L.; Huebl, A.; Almgren, A. Physics of Plasmas, Vol. 28, Issue 2 https://doi.org/10.1063/5.0028512	journal	February 2021
Extended particle absorber for efficient modeling of intense laser–solid interactions Miller, Kyle G.; May, Joshua; Fiuza, Frederico Physics of Plasmas, Vol. 28, Issue 11 https://doi.org/10.1063/5.0065232	journal	November 2021
Interpenetrating Plasma Shells: Near-Equipartition Magnetic Field Generation and Nonthermal Particle Acceleration Silva, L. O.; Fonseca, R. A.; Tonge, J. W. The Astrophysical Journal, Vol. 596, Issue 1 https://doi.org/10.1086/379156	journal	September 2003
Plasma modelling and numerical simulation van Dijk, J.; Kroesen, G. M. W.; Bogaerts, A. Journal of Physics D: Applied Physics, Vol. 42, Issue 19 https://doi.org/10.1088/0022-3727/42/19/190301	journal	September 2009
Exploiting multi-scale parallelism for large scale numerical modelling of laser wakefield accelerators Fonseca, R. A.; Vieira, J.; Fiuza, F. Plasma Physics and Controlled Fusion, Vol. 55, Issue 12 https://doi.org/10.1088/0741-3335/55/12/124011	journal	November 2013
Contemporary particle-in-cell approach to laser-plasma modelling Arber, T. D.; Bennett, K.; Brady, C. S. Plasma Physics and Controlled Fusion, Vol. 57, Issue 11 https://doi.org/10.1088/0741-3335/57/11/113001	journal	September 2015
Numerical heating in particle-in-cell simulations with Monte Carlo binary collisions Alves, E. P.; Mori, W. B.; Fiuza, F. Physical Review E, Vol. 103, Issue 1 https://doi.org/10.1103/PhysRevE.103.013306	journal	January 2021
Spontaneously Growing Transverse Waves in a Plasma Due to an Anisotropic Velocity Distribution Weibel, Erich S. Physical Review Letters, Vol. 2, Issue 3 https://doi.org/10.1103/PhysRevLett.2.83	journal	February 1959
On the energy efficiency of graphics processing units for scientific computing Huang, S.; Xiao, S.; Feng, W. Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5160980	conference	May 2009
Efficient GPU Implementation for Particle in Cell Algorithm Joseph, Rejith George; Ravunnikutty, Girish; Ranka, Sanjay Distributed Processing Symposium (IPDPS), 2011 IEEE International Parallel & Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2011.46	conference	May 2011
GPU Computing Owens, J. D.; Houston, M.; Luebke, D. Proceedings of the IEEE, Vol. 96, Issue 5 https://doi.org/10.1109/JPROC.2008.917757	journal	May 2008
Skeleton Particle-in-Cell Codes on Emerging Computer Architectures Decyk, Viktor K. Computing in Science & Engineering, Vol. 17, Issue 2 https://doi.org/10.1109/MCSE.2014.131	journal	March 2015
Pushing the Frontier in the Design of Laser-Based Electron Accelerators with Groundbreaking Mesh-Refined Particle-In-Cell Simulations on Exascale-Class Supercomputers Fedeli, Luca; Huebl, Axel; Boillod-Cerneux, France SC22: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41404.2022.00008	conference	November 2022
VPIC 2.0: Next Generation Particle-in-Cell Simulations Bird, Robert; Tan, Nigel; Luedtke, Scott V. IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2021.3084795	journal	January 2021
PIConGPU: A Fully Relativistic Particle-in-Cell Code for a GPU Cluster Burau, Heiko; Widera, Renée; Honig, Wolfgang IEEE Transactions on Plasma Science, Vol. 38, Issue 10 https://doi.org/10.1109/TPS.2010.2064310	journal	October 2010
Power Analysis of NERSC Production Workloads Zhao, Zhengji; Rrapaj, Ermal; Bhalachandra, Sridutt Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis https://doi.org/10.1145/3624062.3624200	conference	November 2023
Computer Simulation Using Particles Hockney, R. W.; Eastwood, J. W. https://doi.org/10.1201/9780367806934	book	January 1988

Similar Records

symPACK: A GPU-Capable Fan-Out Sparse Cholesky Solver

Conference · 2023 · OSTI ID:2006802

Bellavita, Julian; Jacquelin, Mathias; Ng, Esmond G; +3 more

Modeling of Ionization Physics with the PIC Code OSIRIS

Conference · 2005 · OSTI ID:878365

Deng, S; Tsung, F; Lee, S; +18 more

A massively parallel time-domain coupled electrodynamics–micromagnetics solver

Journal Article · 2022 · International Journal of High Performance Computing Applications · OSTI ID:1840619

Yao, Zhi; Jambunathan, Revathi; Zeng, Yadong; +1 more

Related Subjects

70 PLASMA PHYSICS AND FUSION TECHNOLOGY
plasma simulation

Title: Acceleration of the particle-in-cell code Osiris with graphics processing units

Citation Formats

References (47)

Similar Records

Related Subjects