A generalized massively parallel ultra-high order FFT-based Maxwell solver

Kallala, Haithem; Vay, Jean-Luc; Vincenti, Henri

doi:10.1016/j.cpc.2019.07.009

Title: A generalized massively parallel ultra-high order FFT-based Maxwell solver

Journal Article · Thu Jul 25 00:00:00 EDT 2019 · Computer Physics Communications

DOI:https://doi.org/10.1016/j.cpc.2019.07.009· OSTI ID:1580963

Kallala, Haithem ^[1]; Vay, Jean-Luc ^[2]; Vincenti, Henri ^[1]

Univ. Paris-Saclay, Gif-sur-Yvette (France)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Dispersion-free ultra-high order FFT-based Maxwell solvers have recently proven to be paramount to a large range of applications, including the high-fidelity modeling of high-intensity laser–matter interactions with Particle-In-Cell (PIC) codes. To enable a massively parallel scaling of these solvers, a novel parallelization technique was recently proposed, which consists in splitting the simulation domain into several processor sub-domains, with guard regions appended at each sub-domain boundary. Maxwell's equations are advanced independently on each sub-domain using local shared-memory FFTs (instead of a single distributed global FFT). This implies small truncation errors at sub-domain boundaries, the amplitude of which depends on guard regions sizes and order of the Maxwell solver. For moderate guard region sizes, this ’local’ technique proved to be highly scalable on up to a million cores and notably enabled the 3D modeling of so-called plasma mirrors, for which 8 guard cells only were enough to prevent truncation error growth. Yet, for other applications, the required number of guard cells might be much higher, which would severely limit the parallel efficiency of this technique due to the large volume of guard cells to be exchanged between sub-domains. In this context, we propose a novel parallelization technique that ensures very good scaling of FFT-based solvers with an arbitrarily high number of guard cells. Our ’hybrid’ technique consists in performing distributed FFTs on local groups of processors with guard regions now appended to boundaries of each group of processors. It uses a dual domain decomposition method for the Maxwell solver and other parts of the PIC cycle to keep the simulation load-balanced. This ’hybrid’ technique was implemented in the open source exascale library PICSAR. Benchmarks show that for a large number of guard cells (>16), the ’hybrid’ technique offers up to ×3 speed-up and ×8 memory savings compared to the ’local’ one.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), High Energy Physics (HEP); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities Division

Grant/Contract Number:: AC02-05CH11231; AC02-06CH11357; 17-SC-20-SC

OSTI ID:: 1580963

Alternate ID(s):: OSTI ID: 1566215

Journal Information:: Computer Physics Communications, Vol. 244, Issue C; ISSN 0010-4655

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 8 works

Citation information provided by
Web of Science

References (17)

Numerical solution of initial boundary value problems involving maxwell's equations in isotropic media IEEE Transactions on Antennas and Propagation, Vol. 14, Issue 3 https://doi.org/10.1109/TAP.1966.1138693	journal	May 1966
Exploiting multi-scale parallelism for large scale numerical modelling of laser wakefield accelerators Fonseca, R. A.; Vieira, J.; Fiuza, F. Plasma Physics and Controlled Fusion, Vol. 55, Issue 12 https://doi.org/10.1088/0741-3335/55/12/124011	journal	November 2013
Pseudospectral Maxwell solvers for an accurate modeling of Doppler harmonic generation on plasma mirrors with particle-in-cell codes Blaclard, G.; Vincenti, H.; Lehe, R. Physical Review E, Vol. 96, Issue 3 https://doi.org/10.1103/PhysRevE.96.033305	journal	September 2017
Ultrahigh-order Maxwell solver with extreme scalability for electromagnetic PIC simulations of plasmas Vincenti, Henri; Vay, Jean-Luc Computer Physics Communications, Vol. 228 https://doi.org/10.1016/j.cpc.2018.03.018	journal	July 2018
Accurate modeling of plasma acceleration with arbitrary order pseudo-spectral particle-in-cell methods Jalas, S.; Dornmair, I.; Lehe, R. Physics of Plasmas, Vol. 24, Issue 3 https://doi.org/10.1063/1.4978569	journal	March 2017
A domain decomposition method for pseudo-spectral electromagnetic simulations of plasmas Vay, Jean-Luc; Haber, Irving; Godfrey, Brendan B. Journal of Computational Physics, Vol. 243 https://doi.org/10.1016/j.jcp.2013.03.010	journal	June 2013
Detailed analysis of the effects of stencil spatial variations with arbitrary high-order finite-difference Maxwell solver Vincenti, H.; Vay, J. -L. Computer Physics Communications, Vol. 200 https://doi.org/10.1016/j.cpc.2015.11.009	journal	March 2016
P3DFFT: A Framework for Parallel Computations of Fourier Transforms in Three Dimensions Pekurovsky, Dmitry SIAM Journal on Scientific Computing, Vol. 34, Issue 4 https://doi.org/10.1137/11082748X	journal	January 2012
Suppressing the numerical Cherenkov instability in FDTD PIC codes Godfrey, Brendan B.; Vay, Jean-Luc Journal of Computational Physics, Vol. 267 https://doi.org/10.1016/j.jcp.2014.02.022	journal	June 2014
An efficient and portable SIMD algorithm for charge/current deposition in Particle-In-Cell codes Vincenti, H.; Lobet, M.; Lehe, R. Computer Physics Communications, Vol. 210 https://doi.org/10.1016/j.cpc.2016.08.023	journal	January 2017
Novel methods in the Particle-In-Cell accelerator Code-Framework Warp Vay, J-L; Grote, D. P.; Cohen, R. H. Computational Science & Discovery, Vol. 5, Issue 1 https://doi.org/10.1088/1749-4699/5/1/014019	journal	January 2012
Smilei : A collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation Derouillat, J.; Beck, A.; Pérez, F. Computer Physics Communications, Vol. 222 https://doi.org/10.1016/j.cpc.2017.09.024	journal	January 2018
Two-step perfectly matched layer for arbitrary-order pseudo-spectral analytical time-domain methods Shapoval, Olga; Vay, Jean-Luc; Vincenti, Henri Computer Physics Communications, Vol. 235 https://doi.org/10.1016/j.cpc.2018.09.015	journal	February 2019
Accurate Modeling Of Plasma Acceleration With Arbitrary Order Pseudo-Spectral Particle-In-Cell Methods Jalas, S.; Dornmair, I.; Lehe, R. Zenodo https://doi.org/10.5281/zenodo.292336	dataset	January 2017
Pseudo-spectral Maxwell solvers for an accurate modeling of Doppler harmonic generation on plasma mirrors with Particle-In-Cell codes Blaclard, G.; Vincenti, H.; Lehe, R. arXiv https://doi.org/10.48550/arxiv.1608.05739	text	January 2016
SMILEI: a collaborative, open-source, multi-purpose particle-in-cell code for plasma simulation Derouillat, J.; Beck, A.; Pérez, F. arXiv https://doi.org/10.48550/arxiv.1702.05128	text	January 2017
Achieving extreme light intensities using relativistic plasma mirrors Vincenti, Henri arXiv https://doi.org/10.48550/arxiv.1812.05357	text	January 2018