Cosmological neutrino simulations at extreme scale
Abstract
Constraining neutrino mass remains an elusive challenge in modern physics. Precision measurements are expected from several upcoming cosmological probes of largescale structure. Achieving this goal relies on an equal level of precision from theoretical predictions of neutrino clustering. Numerical simulations of the nonlinear evolution of cold dark matter and neutrinos play a pivotal role in this process. We incorporate neutrinos into the cosmological Nbody code CUBEP3M and discuss the challenges associated with pushing to the extreme scales demanded by the neutrino problem. We highlight code optimizations made to exploit modern high performance computing architectures and present a novel method of data compression that reduces the phasespace particle footprint from 24 bytes in single precision to roughly 9 bytes. We scale the neutrino problem to the Tianhe2 supercomputer and provide details of our production run, named TianNu, which uses 86% of the machine (13,824 compute nodes). With a total of 2.97 trillion particles, TianNu is currently the world’s largest cosmological Nbody simulation and improves upon previous neutrino simulations by two orders of magnitude in scale. We finish with a discussion of the unanticipated computational challenges that were encountered during the TianNu runtime.
 Authors:
 Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Univ. of Toronto, ON (Canada). Dept. of Astronomy & Astrophysics; Argonne National Lab. (ANL), Argonne, IL (United States)
 Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Peking Univ., Beijing (China). Kavli Inst. for Astronomy & Astrophysics; Beijing Normal Univ., Beijing (China). Dept. of Astronomy
 Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics
 Beijing Normal Univ., Beijing (China). Dept. of Astronomy; Dezhou Univ. (China). Shandong Provincial Key Laboratory of Biophysics
 Univ. of Toronto, ON (Canada). Canadian Inst. for Theoretical Astrophysics; Univ. of Toronto, ON (Canada). Dunlap Inst. for Astronomy and Astrophysics; Canadian Inst. for Advanced Research (Canada); Perimeter Inst. for Theoretical Physics, Waterloo (Canada)
 Univ. of British Columbia, Vancouver, BC (Canada). Dept. of Physics & Astronomy; Univ. of Edinburgh, Scotland (United Kingdom). Scottish University Physics Alliance, Inst. for Astronomy
 Peking Univ., Beijing (China). Dept. of Astronomy
 Beijing Normal Univ., Beijing (China). Dept. of Astronomy
 Chinese Academy of Sciences (CAS), Beijing (China). Key Laboratory for Computational Astrophysics, National Astronomical Observatories
 Chinese Academy of Sciences (CAS), Beijing (China). School of Physical Sciences; Chinese Academy of Sciences (CAS), Beijing (China). Inst. of High Energy Physics
 Publication Date:
 Research Org.:
 Argonne National Lab. (ANL), Argonne, IL (United States)
 Sponsoring Org.:
 Natural Sciences and Engineering Research Council of Canada (NSERC); European Commission (EC); National Science Foundation of China; Fundamental Research Funds for the Central Universities; National Natural Science Foundation of China (NNSFC); Chinese Academy of Sciences (CAS); Ministry of Science and Technology of the People's Republic of China; USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC22)
 OSTI Identifier:
 1393579
 Grant/Contract Number:
 AC0206CH11357
 Resource Type:
 Journal Article: Accepted Manuscript
 Journal Name:
 Research in Astronomy and Astrophysics
 Additional Journal Information:
 Journal Volume: 17; Journal Issue: 8; Journal ID: ISSN 16744527
 Country of Publication:
 United States
 Language:
 English
 Subject:
 79 ASTRONOMY AND ASTROPHYSICS; Cosmology: theory; largescale structure of universe; methods: numerical
Citation Formats
Emberson, J. D., Yu, HaoRan, Inman, Derek, Zhang, TongJie, Pen, UeLi, HarnoisDéraps, Joachim, Yuan, Shuo, Teng, HuanYu, Zhu, HongMing, Chen, Xuelei, and Xing, ZhiZhong. Cosmological neutrino simulations at extreme scale. United States: N. p., 2017.
Web. doi:10.1088/16744527/17/8/85.
Emberson, J. D., Yu, HaoRan, Inman, Derek, Zhang, TongJie, Pen, UeLi, HarnoisDéraps, Joachim, Yuan, Shuo, Teng, HuanYu, Zhu, HongMing, Chen, Xuelei, & Xing, ZhiZhong. Cosmological neutrino simulations at extreme scale. United States. doi:10.1088/16744527/17/8/85.
Emberson, J. D., Yu, HaoRan, Inman, Derek, Zhang, TongJie, Pen, UeLi, HarnoisDéraps, Joachim, Yuan, Shuo, Teng, HuanYu, Zhu, HongMing, Chen, Xuelei, and Xing, ZhiZhong. 2017.
"Cosmological neutrino simulations at extreme scale". United States.
doi:10.1088/16744527/17/8/85.
@article{osti_1393579,
title = {Cosmological neutrino simulations at extreme scale},
author = {Emberson, J. D. and Yu, HaoRan and Inman, Derek and Zhang, TongJie and Pen, UeLi and HarnoisDéraps, Joachim and Yuan, Shuo and Teng, HuanYu and Zhu, HongMing and Chen, Xuelei and Xing, ZhiZhong},
abstractNote = {Constraining neutrino mass remains an elusive challenge in modern physics. Precision measurements are expected from several upcoming cosmological probes of largescale structure. Achieving this goal relies on an equal level of precision from theoretical predictions of neutrino clustering. Numerical simulations of the nonlinear evolution of cold dark matter and neutrinos play a pivotal role in this process. We incorporate neutrinos into the cosmological Nbody code CUBEP3M and discuss the challenges associated with pushing to the extreme scales demanded by the neutrino problem. We highlight code optimizations made to exploit modern high performance computing architectures and present a novel method of data compression that reduces the phasespace particle footprint from 24 bytes in single precision to roughly 9 bytes. We scale the neutrino problem to the Tianhe2 supercomputer and provide details of our production run, named TianNu, which uses 86% of the machine (13,824 compute nodes). With a total of 2.97 trillion particles, TianNu is currently the world’s largest cosmological Nbody simulation and improves upon previous neutrino simulations by two orders of magnitude in scale. We finish with a discussion of the unanticipated computational challenges that were encountered during the TianNu runtime.},
doi = {10.1088/16744527/17/8/85},
journal = {Research in Astronomy and Astrophysics},
number = 8,
volume = 17,
place = {United States},
year = 2017,
month = 8
}
Web of Science

We present a novel, fast and precise method for including the effect of light neutrinos in cosmological Nbody simulations. The effect of the neutrino component is included by using the linear theory neutrino perturbations in the calculation of the gravitational potential in the Nbody simulation. By comparing this new method with the full nonlinear evolution first presented in [1], where the neutrino component was treated as particles, we find that the new method calculates the matter power spectrum with an accuracy better than 1% for {Sigma}m{sub {nu}} {approx}< 0.5 eV at z = 0. This error scales approximately as ({Sigma}m{submore »

The cosmological constant. lambda. and the finescale microwave background anisotropy in a neutrinodominated universe
In a universe with three equally massive neutrino species (meitalic = m/sub ..nu..//sub ..mu../ = m/sub ..nu..//sub tau/ = 30 eV), a cosmological constant ..lambda..not = 0 will diminish the typical correlation scale of finescale angular fluctuations in the microwave background temperature to 3'. The fluctuations will have a dispersion sigma/sub r/roughlyequal2 x 10/sup 5/(1+z/sub 0/) (z/sub 0/ is the redshift when protogalactic irregularities begin to evolve nonlinearly), a value compatible with observational constraints. 
Cosmological constant. lambda. and the finescale microwave background anisotropy in a neutrinodominated universe
In a universe with three equally massive neutrino species (meitalic = m/sub ..nu..//sub ..mu../ = m/sub ..nu..//sub tau/ = 30 eV), a cosmological constant ..lambda..not = 0 will diminish the typical correlation scale of finescale angular flluctuations in the microwave background temperature to 3'. The fluctuations will have a dispersion sigma/sub r/roughlyequal2 x 10/sup 5/(1+z/sub 0/) (z/sub 0/ is the redshift when protogalactic irregularities begin to evolve nonlinearly), a value compatible with observational constraints. 
Cosmological Simulations with ScaleFree Initial Conditions. I. Adiabatic Hydrodynamics
We analyze hierarchical structure formation based on scalefree initial conditions in an Einstein{endash}de Sitter universe, including a baryonic component with {Omega}{sub bary} = 0.05. We present three independent, smoothed particle hydrodynamics (SPH) simulations, performed at two resolutions (32{sup 3} and 64{sup 3} dark matter and baryonic particles) and with two different SPH codes (TreeSPH and P3MSPH). Each simulation is based on identical initial conditions, which consist of Gaussiandistributed initial density fluctuations that have a power spectrum {ital P}({ital k}) {proportional_to} {ital k}{sup {minus}1}. The baryonic material is modeled as an ideal gas subject only to shock heating and adiabatic heatingmore » 
MultiScale Initial Conditions For Cosmological Simulations
We discuss a new algorithm to generate multiscale initial conditions with multiple levels of refinements for cosmological 'zoomin' simulations. The method uses an adaptive convolution of Gaussian white noise with a realspace transfer function kernel together with an adaptive multigrid Poisson solver to generate displacements and velocities following first (1LPT) or secondorder Lagrangian perturbation theory (2LPT). The new algorithm achieves rms relative errors of the order of 10{sup 4} for displacements and velocities in the refinement region and thus improves in terms of errors by about two orders of magnitude over previous approaches. In addition, errors are localized at coarsefinemore »