PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys

Potter, Douglas; Stadel, Joachim; Teyssier, Romain

doi:10.1186/s40668-017-0021-1

Title: PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys

Abstract

We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80 h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18,000 nodes and a peak performance of 10 Pflops.

Authors:

; Stadel, Joachim; Teyssier, Romain

Publication Date:: Thu May 18 00:00:00 EDT 2017

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1357708

Alternate Identifier(s):: OSTI ID: 1567656

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Published Article

Journal Name:: Computational Astrophysics and Cosmology

Additional Journal Information:: Journal Name: Computational Astrophysics and Cosmology Journal Volume: 4 Journal Issue: 1; Journal ID: ISSN 2197-7909

Publisher:: Springer

Country of Publication:: Germany

Language:: English

Subject:: 79 ASTRONOMY AND ASTROPHYSICS; cosmology; astrophysics; simulations

Citation Formats


                    Potter, Douglas, Stadel, Joachim, and Teyssier, Romain. PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys.  Germany: N. p., 2017. 
Web.  doi:10.1186/s40668-017-0021-1.

Copy to clipboard


                    Potter, Douglas, Stadel, Joachim, & Teyssier, Romain. PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys.  Germany.  https://doi.org/10.1186/s40668-017-0021-1

Copy to clipboard


                    Potter, Douglas, Stadel, Joachim, and Teyssier, Romain. Thu .  
"PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys".  Germany.  https://doi.org/10.1186/s40668-017-0021-1.

Copy to clipboard


                    
@article{osti_1357708,

  title        = {PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys},

  author       = {Potter, Douglas and Stadel, Joachim and Teyssier, Romain},

  abstractNote = {We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80 h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18,000 nodes and a peak performance of 10 Pflops.},

  doi          = {10.1186/s40668-017-0021-1},

  journal      = {Computational Astrophysics and Cosmology},

  number       = 1,

  volume       = 4,

  place        = {Germany},

  year         = {Thu May 18 00:00:00 EDT 2017},

  month        = {Thu May 18 00:00:00 EDT 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Publisher's Version of Record
https://doi.org/10.1186/s40668-017-0021-1

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

N-body simulations of gravitational dynamics
journal, May 2011

Dehnen, W.; Read, J. I.
The European Physical Journal Plus, Vol. 126, Issue 5
DOI: 10.1140/epjp/i2011-11055-3

Structure of the Coma Cluster of Galaxies
journal, February 1970

Peebles, P. J. E.
The Astronomical Journal, Vol. 75
DOI: 10.1086/110933

HACC: extreme scaling and performance across diverse architectures
conference, January 2013

Habib, Salman; Morozov, Vitali; Frontiere, Nicholas
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
DOI: 10.1145/2503210.2504566

GreeM: Massively Parallel TreePM Code for Large Cosmological N -body Simulations
journal, December 2009

Ishiyama, Tomoaki; Fukushige, Toshiyuki; Makino, Junichiro
Publications of the Astronomical Society of Japan, Vol. 61, Issue 6
DOI: 10.1093/pasj/61.6.1319

First‐Year Wilkinson Microwave Anisotropy Probe ( WMAP ) Observations: Determination of Cosmological Parameters
journal, September 2003

Spergel, D. N.; Verde, L.; Peiris, H. V.
The Astrophysical Journal Supplement Series, Vol. 148, Issue 1
DOI: 10.1086/377226

Hydra: an Adaptive-Mesh Implementation of P 3M-SPH
journal, October 1995

Couchman, H. M. P.; Thomas, P. A.; Pearce, F. R.
The Astrophysical Journal, Vol. 452
DOI: 10.1086/176348

A parallel hashed Oct-Tree N-body algorithm
conference, January 1993

Warren, M. S.; Salmon, J. K.
Proceedings of the 1993 ACM/IEEE conference on Supercomputing - Supercomputing '93
DOI: 10.1145/169627.169640

Cosmological hydrodynamics with adaptive mesh refinement: A new high resolution code called RAMSES
journal, April 2002

Teyssier, R.
Astronomy & Astrophysics, Vol. 385, Issue 1
DOI: 10.1051/0004-6361:20011817

Computer Simulation Using Particles
book, January 1988

Hockney, R.; Eastwood, J.
CRC Press
DOI: 10.1201/9781439822050

24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs
conference, November 2014

Bedorf, Jeroen; Gaburov, Evghenii; Fujii, Michiko S.
SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2014.10

Planck 2013 results. XVI. Cosmological parameters
journal, October 2014

Ade, P. A. R.; Aghanim, N.; Armitage-Caplan, C.
Astronomy & Astrophysics, Vol. 571
DOI: 10.1051/0004-6361/201321591

A Hierarchical (N) Force Calculation Algorithm
journal, June 2002

Dehnen, Walter
Journal of Computational Physics, Vol. 179, Issue 1
DOI: 10.1006/jcph.2002.7026

First-ever full observable universe simulation
conference, November 2012

Alimi, Jean-Michel; Bouillot, Vincent; Rasera, Yann
2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2012.58

A hierarchical O(N log N) force-calculation algorithm
journal, December 1986

Barnes, Josh; Hut, Piet
Nature, Vol. 324, Issue 6096
DOI: 10.1038/324446a0

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures
journal, January 2016

Habib, Salman; Pope, Adrian; Finkel, Hal
New Astronomy, Vol. 42
DOI: 10.1016/j.newast.2015.06.003

2HOT: an improved parallel hashed oct-tree n-body algorithm for cosmological simulation
conference, January 2013

Warren, Michael S.
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
DOI: 10.1145/2503210.2503220

The cosmological simulation code gadget-2
journal, December 2005

Springel, Volker
Monthly Notices of the Royal Astronomical Society, Vol. 364, Issue 4
DOI: 10.1111/j.1365-2966.2005.09655.x

Matter power spectrum and the challenge of percent accuracy
journal, April 2016

Schneider, Aurel; Teyssier, Romain; Potter, Doug
Journal of Cosmology and Astroparticle Physics, Vol. 2016, Issue 04
DOI: 10.1088/1475-7516/2016/04/047

Towards an accurate mass function for precision cosmology
journal, March 2013

Reed, D. S.; Smith, R. E.; Potter, D.
Monthly Notices of the Royal Astronomical Society, Vol. 431, Issue 2
DOI: 10.1093/mnras/stt301

4.45 Pflops astrophysical N-body simulation on K computer -- The gravitational trillion-body problem
conference, November 2012

Ishiyama, Tomoaki; Nitadori, Keigo; Makino, Junichiro
2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2012.3

On the Clustering Tendencies among the Nebulae. II. a Study of Encounters Between Laboratory Models of Stellar Systems by a New Integration Procedure.
journal, September 1941

Holmberg, Erik
The Astrophysical Journal, Vol. 94
DOI: 10.1086/144344

Application of the Ewald method to cosmological N-body simulations
journal, February 1991

Hernquist, Lars; Bouchet, Francois R.; Suto, Yasushi
The Astrophysical Journal Supplement Series, Vol. 75
DOI: 10.1086/191530

Scaling relations for galaxy clusters in the Millennium-XXL simulation: Scaling relations for clusters in the MXXL
journal, October 2012

Angulo, R. E.; Springel, V.; White, S. D. M.
Monthly Notices of the Royal Astronomical Society, Vol. 426, Issue 3
DOI: 10.1111/j.1365-2966.2012.21830.x

A fast algorithm for particle simulations
journal, December 1987

Greengard, L.; Rokhlin, V.
Journal of Computational Physics, Vol. 73, Issue 2
DOI: 10.1016/0021-9991(87)90140-9

The MICE Grand Challenge light-cone simulation – III. Galaxy lensing mocks from all-sky lensing maps
journal, December 2014

Fosalba, P.; Gaztañaga, E.; Castander, F. J.
Monthly Notices of the Royal Astronomical Society, Vol. 447, Issue 2
DOI: 10.1093/mnras/stu2464

The q Continuum Simulation: Harnessing the Power of gpu Accelerated Supercomputers
journal, August 2015

Heitmann, Katrin; Frontiere, Nicholas; Sewell, Chris
The Astrophysical Journal Supplement Series, Vol. 219, Issue 2
DOI: 10.1088/0067-0049/219/2/34

Similar Records in DOE PAGES and OSTI.GOV collections:

24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs, In: SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Conference Bedorf, Jeroen ; Gaburov, Evghenii ; Fujii, Michiko S. ; ... - SC14: INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS

We have simulated, for the first time, the long term evolution of the Milky Way Galaxy using 51 billion particles on the Swiss Piz Daint supercomputer with our N-body gravitational tree-code Bonsai. Herein, we describe the scientific motivation and numerical algorithms. The Milky Way model was simulated for 6 billion years, during which the bar structure and spiral arms were fully formed. This improves upon previous simulations by using 1000 times more particles, and provides a wealth of new data that can be directly compared with observations. We also report the scalability on both the Swiss Piz Daint and themore »« less
https://doi.org/10.1109/SC.2014.10
Many-Core Acceleration of a Discrete Ordinates Transport Mini-App at Extreme Scale, In: ISC High Performance 2016: High Performance Computing

Conference Deakin, Tom ; McIntosh-Smith, Simon ; Gaudin, Wayne - HIGH PERFORMANCE COMPUTING

Time-dependent deterministic discrete ordinates transport codes are an important class of application which provide significant challenges for large, many-core systems. One such challenge is the large memory capacity needed by the solve step, which requires us to have a scalable solution in order to have enough node-level memory to store all the data. In our previous work, we demonstrated the first implementation which showed a significant performance benefit for single node solves using GPUs. In this paper we extend our work to large problems and demonstrate the scalability of our solution on two Petascale GPU-based supercomputers: Titan at Oak Ridgemore »« less
https://doi.org/10.1007/978-3-319-41321-1_22
HACC: extreme scaling and performance across diverse architectures, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis

Journal Article Habib, Salman ; Morozov, Vitali ; Frontiere, Nicholas ; ... - International Conference for High Performance Computing, Networking, Storage and Analysis

Supercomputing is evolving towards hybrid and accelerator-based architectures with millions of cores. The HACC (Hardware/Hybrid Accelerated Cosmology Code) framework exploits this diverse landscape at the largest scales of problem size, obtaining high scalability and sustained performance. Developed to satisfy the science requirements of cosmological surveys, HACC melds particle and grid methods using a novel algorithmic structure that flexibly maps across architectures, including CPU/GPU, multi/many-core, and Blue Gene systems. We demonstrate the success of HACC on two very different machines, the CPU/GPU system Titan and the BG/Q systems Sequoia and Mira, attaining unprecedented levels of scalable performance. We demonstrate strong andmore »« less
https://doi.org/10.1145/2503210.2504566
HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Journal Article Habib, Salman ; Pope, Adrian ; Finkel, Hal ; ... - New Astronomy

Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers thatmore »« less
https://doi.org/10.1016/j.newast.2015.06.003
HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Journal Article Habib, Salman ; Pope, Adrian ; Finkel, Hal ; ... - New Astronomy

Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers thatmore »« less
Cited by 139
https://doi.org/10.1016/j.newast.2015.06.003

Full Text Available

Similar Records

Title: PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys

Abstract

Citation Formats

N-body simulations of gravitational dynamics journal, May 2011

Structure of the Coma Cluster of Galaxies journal, February 1970

HACC: extreme scaling and performance across diverse architectures conference, January 2013

GreeM: Massively Parallel TreePM Code for Large Cosmological N -body Simulations journal, December 2009

First‐Year Wilkinson Microwave Anisotropy Probe ( WMAP ) Observations: Determination of Cosmological Parameters journal, September 2003

Hydra: an Adaptive-Mesh Implementation of P 3M-SPH journal, October 1995

A parallel hashed Oct-Tree N-body algorithm conference, January 1993

Cosmological hydrodynamics with adaptive mesh refinement: A new high resolution code called RAMSES journal, April 2002

Computer Simulation Using Particles book, January 1988

24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs conference, November 2014

Planck 2013 results. XVI. Cosmological parameters journal, October 2014

A Hierarchical (N) Force Calculation Algorithm journal, June 2002

First-ever full observable universe simulation conference, November 2012

A hierarchical O(N log N) force-calculation algorithm journal, December 1986

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures journal, January 2016

2HOT: an improved parallel hashed oct-tree n-body algorithm for cosmological simulation conference, January 2013

The cosmological simulation code gadget-2 journal, December 2005

Matter power spectrum and the challenge of percent accuracy journal, April 2016

Towards an accurate mass function for precision cosmology journal, March 2013

4.45 Pflops astrophysical N-body simulation on K computer -- The gravitational trillion-body problem conference, November 2012

On the Clustering Tendencies among the Nebulae. II. a Study of Encounters Between Laboratory Models of Stellar Systems by a New Integration Procedure. journal, September 1941

Application of the Ewald method to cosmological N-body simulations journal, February 1991

Scaling relations for galaxy clusters in the Millennium-XXL simulation: Scaling relations for clusters in the MXXL journal, October 2012

A fast algorithm for particle simulations journal, December 1987

The MICE Grand Challenge light-cone simulation – III. Galaxy lensing mocks from all-sky lensing maps journal, December 2014

The q Continuum Simulation: Harnessing the Power of gpu Accelerated Supercomputers journal, August 2015

N-body simulations of gravitational dynamics
journal, May 2011

Structure of the Coma Cluster of Galaxies
journal, February 1970

HACC: extreme scaling and performance across diverse architectures
conference, January 2013

GreeM: Massively Parallel TreePM Code for Large Cosmological N -body Simulations
journal, December 2009

First‐Year Wilkinson Microwave Anisotropy Probe ( WMAP ) Observations: Determination of Cosmological Parameters
journal, September 2003

Hydra: an Adaptive-Mesh Implementation of P 3M-SPH
journal, October 1995

A parallel hashed Oct-Tree N-body algorithm
conference, January 1993

Cosmological hydrodynamics with adaptive mesh refinement: A new high resolution code called RAMSES
journal, April 2002

Computer Simulation Using Particles
book, January 1988

24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs
conference, November 2014

Planck 2013 results. XVI. Cosmological parameters
journal, October 2014

A Hierarchical (N) Force Calculation Algorithm
journal, June 2002

First-ever full observable universe simulation
conference, November 2012

A hierarchical O(N log N) force-calculation algorithm
journal, December 1986

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures
journal, January 2016

2HOT: an improved parallel hashed oct-tree n-body algorithm for cosmological simulation
conference, January 2013

The cosmological simulation code gadget-2
journal, December 2005

Matter power spectrum and the challenge of percent accuracy
journal, April 2016

Towards an accurate mass function for precision cosmology
journal, March 2013

4.45 Pflops astrophysical N-body simulation on K computer -- The gravitational trillion-body problem
conference, November 2012

On the Clustering Tendencies among the Nebulae. II. a Study of Encounters Between Laboratory Models of Stellar Systems by a New Integration Procedure.
journal, September 1941

Application of the Ewald method to cosmological N-body simulations
journal, February 1991

Scaling relations for galaxy clusters in the Millennium-XXL simulation: Scaling relations for clusters in the MXXL
journal, October 2012

A fast algorithm for particle simulations
journal, December 1987

The MICE Grand Challenge light-cone simulation – III. Galaxy lensing mocks from all-sky lensing maps
journal, December 2014

The q Continuum Simulation: Harnessing the Power of gpu Accelerated Supercomputers
journal, August 2015