skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys

Abstract

We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80 h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18,000 nodes and a peak performance of 10 Pflops.

Authors:
ORCiD logo; ;
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1357708
Alternate Identifier(s):
OSTI ID: 1567656
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Published Article
Journal Name:
Computational Astrophysics and Cosmology
Additional Journal Information:
Journal Name: Computational Astrophysics and Cosmology Journal Volume: 4 Journal Issue: 1; Journal ID: ISSN 2197-7909
Publisher:
Springer
Country of Publication:
Germany
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; cosmology; astrophysics; simulations

Citation Formats

Potter, Douglas, Stadel, Joachim, and Teyssier, Romain. PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys. Germany: N. p., 2017. Web. doi:10.1186/s40668-017-0021-1.
Potter, Douglas, Stadel, Joachim, & Teyssier, Romain. PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys. Germany. doi:10.1186/s40668-017-0021-1.
Potter, Douglas, Stadel, Joachim, and Teyssier, Romain. Thu . "PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys". Germany. doi:10.1186/s40668-017-0021-1.
@article{osti_1357708,
title = {PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys},
author = {Potter, Douglas and Stadel, Joachim and Teyssier, Romain},
abstractNote = {We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80 h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18,000 nodes and a peak performance of 10 Pflops.},
doi = {10.1186/s40668-017-0021-1},
journal = {Computational Astrophysics and Cosmology},
number = 1,
volume = 4,
place = {Germany},
year = {2017},
month = {5}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
DOI: 10.1186/s40668-017-0021-1

Save / Share:

Works referenced in this record:

N-body simulations of gravitational dynamics
journal, May 2011


Structure of the Coma Cluster of Galaxies
journal, February 1970

  • Peebles, P. J. E.
  • The Astronomical Journal, Vol. 75
  • DOI: 10.1086/110933

HACC: extreme scaling and performance across diverse architectures
conference, January 2013

  • Habib, Salman; Morozov, Vitali; Frontiere, Nicholas
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
  • DOI: 10.1145/2503210.2504566

GreeM: Massively Parallel TreePM Code for Large Cosmological N -body Simulations
journal, December 2009

  • Ishiyama, Tomoaki; Fukushige, Toshiyuki; Makino, Junichiro
  • Publications of the Astronomical Society of Japan, Vol. 61, Issue 6
  • DOI: 10.1093/pasj/61.6.1319

First‐Year Wilkinson Microwave Anisotropy Probe ( WMAP ) Observations: Determination of Cosmological Parameters
journal, September 2003

  • Spergel, D. N.; Verde, L.; Peiris, H. V.
  • The Astrophysical Journal Supplement Series, Vol. 148, Issue 1
  • DOI: 10.1086/377226

Hydra: an Adaptive-Mesh Implementation of P 3M-SPH
journal, October 1995

  • Couchman, H. M. P.; Thomas, P. A.; Pearce, F. R.
  • The Astrophysical Journal, Vol. 452
  • DOI: 10.1086/176348

A parallel hashed Oct-Tree N-body algorithm
conference, January 1993

  • Warren, M. S.; Salmon, J. K.
  • Proceedings of the 1993 ACM/IEEE conference on Supercomputing - Supercomputing '93
  • DOI: 10.1145/169627.169640

Cosmological hydrodynamics with adaptive mesh refinement: A new high resolution code called RAMSES
journal, April 2002


24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs
conference, November 2014

  • Bedorf, Jeroen; Gaburov, Evghenii; Fujii, Michiko S.
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2014.10

Planck 2013 results. XVI. Cosmological parameters
journal, October 2014


A Hierarchical (N) Force Calculation Algorithm
journal, June 2002


A hierarchical O(N log N) force-calculation algorithm
journal, December 1986


HACC: Simulating sky surveys on state-of-the-art supercomputing architectures
journal, January 2016


2HOT: an improved parallel hashed oct-tree n-body algorithm for cosmological simulation
conference, January 2013

  • Warren, Michael S.
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
  • DOI: 10.1145/2503210.2503220

The cosmological simulation code gadget-2
journal, December 2005


Matter power spectrum and the challenge of percent accuracy
journal, April 2016

  • Schneider, Aurel; Teyssier, Romain; Potter, Doug
  • Journal of Cosmology and Astroparticle Physics, Vol. 2016, Issue 04
  • DOI: 10.1088/1475-7516/2016/04/047

Towards an accurate mass function for precision cosmology
journal, March 2013

  • Reed, D. S.; Smith, R. E.; Potter, D.
  • Monthly Notices of the Royal Astronomical Society, Vol. 431, Issue 2
  • DOI: 10.1093/mnras/stt301

4.45 Pflops astrophysical N-body simulation on K computer -- The gravitational trillion-body problem
conference, November 2012

  • Ishiyama, Tomoaki; Nitadori, Keigo; Makino, Junichiro
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2012.3

Application of the Ewald method to cosmological N-body simulations
journal, February 1991

  • Hernquist, Lars; Bouchet, Francois R.; Suto, Yasushi
  • The Astrophysical Journal Supplement Series, Vol. 75
  • DOI: 10.1086/191530

Scaling relations for galaxy clusters in the Millennium-XXL simulation: Scaling relations for clusters in the MXXL
journal, October 2012


A fast algorithm for particle simulations
journal, December 1987


The MICE Grand Challenge light-cone simulation – III. Galaxy lensing mocks from all-sky lensing maps
journal, December 2014

  • Fosalba, P.; Gaztañaga, E.; Castander, F. J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 447, Issue 2
  • DOI: 10.1093/mnras/stu2464

The q Continuum Simulation: Harnessing the Power of gpu Accelerated Supercomputers
journal, August 2015

  • Heitmann, Katrin; Frontiere, Nicholas; Sewell, Chris
  • The Astrophysical Journal Supplement Series, Vol. 219, Issue 2
  • DOI: 10.1088/0067-0049/219/2/34

    Works referencing / citing this record:

    Interactive 3D visualization for theoretical virtual observatories
    journal, April 2018

    • Dykes, T.; Hassan, A.; Gheller, C.
    • Monthly Notices of the Royal Astronomical Society, Vol. 477, Issue 2
    • DOI: 10.1093/mnras/sty855