skip to main content

DOE PAGESDOE PAGES

Title: Cosmological neutrino simulations at extreme scale

Constraining neutrino mass remains an elusive challenge in modern physics. Precision measurements are expected from several upcoming cosmological probes of large-scale structure. Achieving this goal relies on an equal level of precision from theoretical predictions of neutrino clustering. Numerical simulations of the non-linear evolution of cold dark matter and neutrinos play a pivotal role in this process. We incorporate neutrinos into the cosmological N-body code CUBEP3M and discuss the challenges associated with pushing to the extreme scales demanded by the neutrino problem. We highlight code optimizations made to exploit modern high performance computing architectures and present a novel method of data compression that reduces the phase-space particle footprint from 24 bytes in single precision to roughly 9 bytes. We scale the neutrino problem to the Tianhe-2 supercomputer and provide details of our production run, named TianNu, which uses 86% of the machine (13,824 compute nodes). With a total of 2.97 trillion particles, TianNu is currently the world’s largest cosmological N-body simulation and improves upon previous neutrino simulations by two orders of magnitude in scale. We finish with a discussion of the unanticipated computational challenges that were encountered during the TianNu runtime.
Authors:
 [1] ;  [2] ;  [3] ;  [4] ;  [5] ;  [6] ;  [7] ;  [8] ;  [9] ;  [9] ;  [10]
  1. Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Univ. of Toronto, ON (Canada). Dept. of Astronomy & Astrophysics; Argonne National Lab. (ANL), Argonne, IL (United States)
  2. Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Peking Univ., Beijing (China). Kavli Inst. for Astronomy & Astrophysics; Beijing Normal Univ., Beijing (China). Dept. of Astronomy
  3. Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics
  4. Beijing Normal Univ., Beijing (China). Dept. of Astronomy; Dezhou Univ. (China). Shandong Provincial Key Laboratory of Biophysics
  5. Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Univ. of Toronto, ON (Canada). Dunlap Inst. for Astronomy and Astrophysics; Canadian Inst. for Advanced Research (Canada); Perimeter Inst. for Theoretical Physics, Waterloo (Canada)
  6. Univ. of British Columbia, Vancouver, BC (Canada). Dept. of Physics & Astronomy; Univ. of Edinburgh, Scotland (United Kingdom). Scottish University Physics Alliance, Inst. for Astronomy
  7. Peking Univ., Beijing (China). Dept. of Astronomy
  8. Beijing Normal Univ., Beijing (China). Dept. of Astronomy
  9. Chinese Academy of Sciences (CAS), Beijing (China). Key Laboratory for Computational Astrophysics, National Astronomical Observatories
  10. Chinese Academy of Sciences (CAS), Beijing (China). School of Physical Sciences; Chinese Academy of Sciences (CAS), Beijing (China). Inst. of High Energy Physics
Publication Date:
Grant/Contract Number:
AC02-06CH11357
Type:
Accepted Manuscript
Journal Name:
Research in Astronomy and Astrophysics
Additional Journal Information:
Journal Volume: 17; Journal Issue: 8; Journal ID: ISSN 1674-4527
Research Org:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org:
Natural Sciences and Engineering Research Council of Canada (NSERC); European Commission (EC); National Science Foundation of China; Fundamental Research Funds for the Central Universities; National Natural Science Foundation of China (NNSFC); Chinese Academy of Sciences (CAS); Ministry of Science and Technology of the People's Republic of China; USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; Cosmology: theory; large-scale structure of universe; methods: numerical
OSTI Identifier:
1393579

Emberson, J. D., Yu, Hao-Ran, Inman, Derek, Zhang, Tong-Jie, Pen, Ue-Li, Harnois-Déraps, Joachim, Yuan, Shuo, Teng, Huan-Yu, Zhu, Hong-Ming, Chen, Xuelei, and Xing, Zhi-Zhong. Cosmological neutrino simulations at extreme scale. United States: N. p., Web. doi:10.1088/1674-4527/17/8/85.
Emberson, J. D., Yu, Hao-Ran, Inman, Derek, Zhang, Tong-Jie, Pen, Ue-Li, Harnois-Déraps, Joachim, Yuan, Shuo, Teng, Huan-Yu, Zhu, Hong-Ming, Chen, Xuelei, & Xing, Zhi-Zhong. Cosmological neutrino simulations at extreme scale. United States. doi:10.1088/1674-4527/17/8/85.
Emberson, J. D., Yu, Hao-Ran, Inman, Derek, Zhang, Tong-Jie, Pen, Ue-Li, Harnois-Déraps, Joachim, Yuan, Shuo, Teng, Huan-Yu, Zhu, Hong-Ming, Chen, Xuelei, and Xing, Zhi-Zhong. 2017. "Cosmological neutrino simulations at extreme scale". United States. doi:10.1088/1674-4527/17/8/85. https://www.osti.gov/servlets/purl/1393579.
@article{osti_1393579,
title = {Cosmological neutrino simulations at extreme scale},
author = {Emberson, J. D. and Yu, Hao-Ran and Inman, Derek and Zhang, Tong-Jie and Pen, Ue-Li and Harnois-Déraps, Joachim and Yuan, Shuo and Teng, Huan-Yu and Zhu, Hong-Ming and Chen, Xuelei and Xing, Zhi-Zhong},
abstractNote = {Constraining neutrino mass remains an elusive challenge in modern physics. Precision measurements are expected from several upcoming cosmological probes of large-scale structure. Achieving this goal relies on an equal level of precision from theoretical predictions of neutrino clustering. Numerical simulations of the non-linear evolution of cold dark matter and neutrinos play a pivotal role in this process. We incorporate neutrinos into the cosmological N-body code CUBEP3M and discuss the challenges associated with pushing to the extreme scales demanded by the neutrino problem. We highlight code optimizations made to exploit modern high performance computing architectures and present a novel method of data compression that reduces the phase-space particle footprint from 24 bytes in single precision to roughly 9 bytes. We scale the neutrino problem to the Tianhe-2 supercomputer and provide details of our production run, named TianNu, which uses 86% of the machine (13,824 compute nodes). With a total of 2.97 trillion particles, TianNu is currently the world’s largest cosmological N-body simulation and improves upon previous neutrino simulations by two orders of magnitude in scale. We finish with a discussion of the unanticipated computational challenges that were encountered during the TianNu runtime.},
doi = {10.1088/1674-4527/17/8/85},
journal = {Research in Astronomy and Astrophysics},
number = 8,
volume = 17,
place = {United States},
year = {2017},
month = {8}
}