PKDGRAV3: beyond trillion particle cosmological simulations for the next era of galaxy surveys
We report on the successful completion of a 2 trillion particle cosmological simulation to z=0 run on the Piz Daint supercomputer (CSCS, Switzerland), using 4000+ GPU nodes for a little less than 80 h of wall-clock time or 350,000 node hours. Using multiple benchmarks and performance measurements on the US Oak Ridge National Laboratory Titan supercomputer, we demonstrate that our code PKDGRAV3, delivers, to our knowledge, the fastest time-to-solution for large-scale cosmological N-body simulations. This was made possible by using the Fast Multipole Method in conjunction with individual and adaptive particle time steps, both deployed efficiently (and for the first time) on supercomputers with GPU-accelerated nodes. The very low memory footprint of PKDGRAV3 allowed us to run the first ever benchmark with 8 trillion particles on Titan, and to achieve perfect scaling up to 18,000 nodes and a peak performance of 10 Pflops.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- Grant/Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1357708
- Alternate ID(s):
- OSTI ID: 1567656
- Journal Information:
- Computational Astrophysics and Cosmology, Journal Name: Computational Astrophysics and Cosmology Vol. 4 Journal Issue: 1; ISSN 2197-7909
- Publisher:
- SpringerCopyright Statement
- Country of Publication:
- Germany
- Language:
- English
Similar Records
Many-Core Acceleration of a Discrete Ordinates Transport Mini-App at Extreme Scale, In: ISC High Performance 2016: High Performance Computing
HACC: extreme scaling and performance across diverse architectures, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis