skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms

Abstract

The Gyrokinetic Toroidal Code (GTC) uses the particle-in-cell method to efficiently simulate plasma microturbulence. This paper presents novel analysis and optimization techniques to enhance the performance of GTC on large-scale machines. We introduce cell access analysis to better manage locality vs. synchronization tradeoffs on CPU and GPU-based architectures. Finally, our optimized hybrid parallel implementation of GTC uses MPI, OpenMP, and NVIDIA CUDA, achieves up to a 2× speedup over the reference Fortran version on multiple parallel systems, and scales efficiently to tens of thousands of cores.

Authors:
 [1];  [2];  [1];  [3];  [4];  [1]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  2. Pennsylvania State Univ., University Park, PA (United States)
  3. Princeton Univ., NJ (United States). Princeton Inst. of Computational Science and Engineering
  4. Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1407202
Grant/Contract Number:  
AC02-05CH11231; AC02-09CH11466; AC02-06CH11357; AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of High Performance Computing Applications
Additional Journal Information:
Journal Volume: 27; Journal Issue: 4; Journal ID: ISSN 1094-3420
Publisher:
SAGE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, Wang, Bei, Ethier, Stephane, and Oliker, Leonid. Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms. United States: N. p., 2013. Web. doi:10.1177/1094342013492446.
Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, Wang, Bei, Ethier, Stephane, & Oliker, Leonid. Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms. United States. doi:10.1177/1094342013492446.
Ibrahim, Khaled Z., Madduri, Kamesh, Williams, Samuel, Wang, Bei, Ethier, Stephane, and Oliker, Leonid. Thu . "Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms". United States. doi:10.1177/1094342013492446. https://www.osti.gov/servlets/purl/1407202.
@article{osti_1407202,
title = {Analysis and optimization of gyrokinetic toroidal simulations on homogenous and heterogenous platforms},
author = {Ibrahim, Khaled Z. and Madduri, Kamesh and Williams, Samuel and Wang, Bei and Ethier, Stephane and Oliker, Leonid},
abstractNote = {The Gyrokinetic Toroidal Code (GTC) uses the particle-in-cell method to efficiently simulate plasma microturbulence. This paper presents novel analysis and optimization techniques to enhance the performance of GTC on large-scale machines. We introduce cell access analysis to better manage locality vs. synchronization tradeoffs on CPU and GPU-based architectures. Finally, our optimized hybrid parallel implementation of GTC uses MPI, OpenMP, and NVIDIA CUDA, achieves up to a 2× speedup over the reference Fortran version on multiple parallel systems, and scales efficiently to tens of thousands of cores.},
doi = {10.1177/1094342013492446},
journal = {International Journal of High Performance Computing Applications},
number = 4,
volume = 27,
place = {United States},
year = {2013},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 7 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Adaptable Particle-in-Cell algorithms for graphical processing units
journal, March 2011


0.374 Pflop/s trillion-particle kinetic modeling of laser plasma interaction on roadrunner
conference, November 2008

  • Bowers, K. J.; Albright, B. J.; Bergen, B.
  • 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2008.5222734

Method to increase the simulation speed of particle-in-cell (PIC) code
journal, December 2001


Cosmological N-Body Simulations
journal, January 1991

  • Bertschinger, Edmund; Gelb, James M.
  • Computers in Physics, Vol. 5, Issue 2
  • DOI: 10.1063/1.4822978

Scientific Computations on Modern Parallel Vector Systems
conference, January 2004

  • Oliker, L.; Canning, A.; Carter, J.
  • Proceedings of the ACM/IEEE SC2004 Conference
  • DOI: 10.1109/SC.2004.54

A finite element Poisson solver for gyrokinetic particle simulations in a global field aligned mesh
journal, May 2006

  • Nishimura, Y.; Lin, Z.; Lewandowski, J. L. V.
  • Journal of Computational Physics, Vol. 214, Issue 2
  • DOI: 10.1016/j.jcp.2005.10.011

Gyrokinetic particle simulation model
journal, September 1987


OhHelp: a scalable domain-decomposing dynamic load balancing for particle-in-cell simulations
conference, January 2009

  • Nakashima, Hiroshi; Miyake, Yohei; Usui, Hideyuki
  • Proceedings of the 23rd international conference on Conference on Supercomputing - ICS '09
  • DOI: 10.1145/1542275.1542293

QUICKPIC: A highly efficient particle-in-cell code for modeling wakefield acceleration in plasmas
journal, September 2006


Fast parallel Particle-To-Grid interpolation for plasma PIC simulations on the GPU
journal, October 2008

  • Stantchev, George; Dorland, William; Gumerov, Nail
  • Journal of Parallel and Distributed Computing, Vol. 68, Issue 10
  • DOI: 10.1016/j.jpdc.2008.05.009

Particle-in-cell simulation codes in High Performance Fortran
conference, January 1996

  • Akarsu, Erol; Dincer, Kivanc; Haupt, Tomasz
  • Proceedings of the 1996 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '96
  • DOI: 10.1145/369028.369108

Turbulent Transport Reduction by Zonal Flows: Massively Parallel Simulations
journal, September 1998


Gyrokinetic toroidal simulations on leading multi- and manycore HPC systems
conference, January 2011

  • Madduri, Kamesh; Ibrahim, Khaled Z.; Williams, Samuel
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11
  • DOI: 10.1145/2063384.2063415

On parallel PIC versatility and the structure of parallel PIC approaches
journal, December 1997


Large-scale gyrokinetic particle simulation of microturbulence in magnetically confined fusion plasmas
journal, January 2008

  • Ethier, S.; Tang, W. M.; Walkup, R.
  • IBM Journal of Research and Development, Vol. 52, Issue 1.2
  • DOI: 10.1147/rd.521.0105

Method for solving the gyrokinetic Poisson equation in general geometry
journal, November 1995


UPIC: A framework for massively parallel particle-in-cell codes
journal, July 2007


Accelerating a Particle-in-Cell Simulation Using a Hybrid Counting Sort
journal, November 2001


VORPAL: a versatile plasma simulation code
journal, May 2004


Memory-efficient optimization of Gyrokinetic particle-to-grid interpolation for multicore processors
conference, January 2009

  • Madduri, Kamesh; Williams, Samuel; Ethier, Stéphane
  • Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09
  • DOI: 10.1145/1654059.1654108