skip to main content


Title: In situ and in-transit analysis of cosmological simulations

Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent ‘offline’ analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid MPI+OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own (‘ in situ’). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other ‘sidecar’ group, which post-processes it while the simulation continues (‘in-transit’). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge treesmore » to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.« less
ORCiD logo [1] ;  [1] ;  [1] ;  [1] ;  [1] ;  [1] ;  [1]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Publication Date:
Report Number(s):
Journal ID: ISSN 2197-7909; ir:1006104
Grant/Contract Number:
Published Article
Journal Name:
Computational Astrophysics and Cosmology
Additional Journal Information:
Journal Volume: 3; Journal Issue: 1; Journal ID: ISSN 2197-7909
Research Org:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Country of Publication:
United States
79 ASTRONOMY AND ASTROPHYSICS; cosmology; post-processing; halo-finding; power spectra; in situ in-transit
OSTI Identifier:
Alternate Identifier(s):
OSTI ID: 1342534; OSTI ID: 1379575