skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers

Abstract

The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and particle decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method to extreme computational scales. In this paper, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputingmore » systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects and network topologies. Furthermore, these performance comparisons using a realistic discovery-science-capable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide.« less

Authors:
 [1];  [2];  [3];  [4];  [5];  [4];  [4]
  1. Princeton Univ., Princeton, NJ (United States)
  2. Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States)
  3. Princeton Univ., Princeton, NJ (United States); Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States)
  4. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  5. Pennsylvania State Univ., University Park, PA (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory, Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1398471
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of High Performance Computing Applications
Additional Journal Information:
Journal Volume: 33; Journal Issue: 1; Journal ID: ISSN 1094-3420
Publisher:
SAGE
Country of Publication:
United States
Language:
English
Subject:
70 PLASMA PHYSICS AND FUSION TECHNOLOGY; 97 MATHEMATICS AND COMPUTING; Particle-in-cell methods; Vlasov–Poisson equations; NVIDIA GPU; Intel Xeon Phi (MIC); heterogeneous systems; fusion plasma simulations; extreme scale

Citation Formats

Wang, Bei, Ethier, Stephane, Tang, William, Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, and Oliker, Leonid. Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers. United States: N. p., 2017. Web. doi:10.1177/1094342017712059.
Wang, Bei, Ethier, Stephane, Tang, William, Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, & Oliker, Leonid. Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers. United States. doi:10.1177/1094342017712059.
Wang, Bei, Ethier, Stephane, Tang, William, Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, and Oliker, Leonid. Thu . "Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers". United States. doi:10.1177/1094342017712059. https://www.osti.gov/servlets/purl/1398471.
@article{osti_1398471,
title = {Modern gyrokinetic particle-in-cell simulation of fusion plasmas on top supercomputers},
author = {Wang, Bei and Ethier, Stephane and Tang, William and Ibrahim, Khaled Z. and Madduri, Kamesh and Williams, Samuel and Oliker, Leonid},
abstractNote = {The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and particle decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method to extreme computational scales. In this paper, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputing systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects and network topologies. Furthermore, these performance comparisons using a realistic discovery-science-capable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide.},
doi = {10.1177/1094342017712059},
journal = {International Journal of High Performance Computing Applications},
number = 1,
volume = 33,
place = {United States},
year = {2017},
month = {6}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 1 work
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: An illustration of 3D toroidal grid.

Save / Share:

Works referenced in this record:

A generalized weight-based particle-in-cell simulation scheme
journal, March 2011


A unified Monte Carlo interpretation of particle simulations and applications to non‐neutral plasmas
journal, April 1994


A Particle-in-cell Method with Adaptive Phase-space Remapping for Kinetic Plasmas
journal, January 2011

  • Wang, B.; Miller, G. H.; Colella, P.
  • SIAM Journal on Scientific Computing, Vol. 33, Issue 6
  • DOI: 10.1137/100811805

Long global gyrokinetic simulations: Source terms and particle noise control
journal, May 2008

  • McMillan, B. F.; Jolliet, S.; Tran, T. M.
  • Physics of Plasmas, Vol. 15, Issue 5
  • DOI: 10.1063/1.2921792

Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms
journal, July 2013

  • Ibrahim, Khaled Z.; Madduri, Kamesh; Williams, Samuel
  • The International Journal of High Performance Computing Applications, Vol. 27, Issue 4
  • DOI: 10.1177/1094342013492446

An Adaptive, High-Order Phase-Space Remapping for the Two Dimensional Vlasov--Poisson Equations
journal, January 2012

  • Wang, Bei; Miller, Greg; Colella, Phil
  • SIAM Journal on Scientific Computing, Vol. 34, Issue 6
  • DOI: 10.1137/120872954

Coarse-graining phase space in δf particle-in-cell simulations
journal, August 2007

  • Chen, Yang; Parker, Scott E.
  • Physics of Plasmas, Vol. 14, Issue 8
  • DOI: 10.1063/1.2751603

A global collisionless PIC code in magnetic coordinates
journal, September 2007


Scientific Discovery in Fusion Plasma Turbulence Simulations at Extreme Scale
journal, September 2014

  • Tang, William; Wang, Bei; Ethier, Stephane
  • Computing in Science & Engineering, Vol. 16, Issue 5
  • DOI: 10.1109/MCSE.2014.54

Plasma size and power scaling of ion temperature gradient driven turbulence
journal, February 2014

  • Idomura, Yasuhiro; Nakata, Motoki
  • Physics of Plasmas, Vol. 21, Issue 2
  • DOI: 10.1063/1.4867379

Gyrokinetic simulation of ion temperature gradient driven turbulence in 3D toroidal geometry
journal, September 1993


Gyrokinetic particle simulation model
journal, September 1987


Turbulent Transport Reduction by Zonal Flows: Massively Parallel Simulations
journal, September 1998


Large-scale gyrokinetic particle simulation of microturbulence in magnetically confined fusion plasmas
journal, January 2008

  • Ethier, S.; Tang, W. M.; Walkup, R.
  • IBM Journal of Research and Development, Vol. 52, Issue 1.2
  • DOI: 10.1147/rd.521.0105

Method for solving the gyrokinetic Poisson equation in general geometry
journal, November 1995


System Size Effects on Gyrokinetic Turbulence
journal, October 2010


Comparisons and physics basis of tokamak transport models and turbulence simulations
journal, March 2000

  • Dimits, A. M.; Bateman, G.; Beer, M. A.
  • Physics of Plasmas, Vol. 7, Issue 3
  • DOI: 10.1063/1.873896

Gyrokinetic particle-in-cell simulations of plasma microturbulence on advanced computing platforms
journal, January 2005


Size Scaling of Turbulent Transport in Magnetically Confined Plasmas
journal, April 2002