Title: Scaling pair count to next galaxy surveys

Abstract

ABSTRACT Counting pairs of galaxies or stars according to their distance is at the core of real-space correlation analyses performed in astrophysics and cosmology. Upcoming galaxy surveys (LSST, Euclid) will measure properties of billions of galaxies challenging our ability to perform such counting in a minute-scale time relevant for the usage of simulations. The problem is only limited by efficient access to the data, hence belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of non-hierarchical sphere pixelization based on cube symmetries and develop a new one dubbed the ‘Similar Radius Sphere Pixelization’ (SARSPix) with very close to square pixels. It provides the most adapted indexing over the sphere for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In each case, we achieve the construction of a standard pair-distance histogram in about 2 min, using a simple algorithm that is shown to scale, over a moderate number of nodes (16–64). This illustrates the potentialmore » of this new techniques in the field of astronomy where data access is becoming the main bottleneck. They can be easily adapted to other use-cases as nearest-neighbours search, catalogue cross-match or cluster finding. The software is publicly available from https://github.com/astrolabsoftware/SparkCorr.« less

Authors:

Plaszczynski, S. ;

• Search DOE PAGES for author "Plaszczynski, S."
• Search DOE PAGES for ORCID "0000-0002-1278-109X"

Campagne, J. E. ;

• Search DOE PAGES for author "Campagne, J. E."
• Search DOE PAGES for ORCID "0000-0002-1590-6927"

Peloton, J. ;

• Search DOE PAGES for author "Peloton, J."
• Search DOE PAGES for ORCID "0000-0002-8560-4449"

Arnault, C.

Publication Date:

Wed Dec 15 00:00:00 EST 2021

Research Org.:

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Org.:

USDOE Office of Science (SC)

OSTI Identifier:

1839314

Alternate Identifier(s):

OSTI ID: 1982573

Grant/Contract Number:  

AC02-05CH11231

Resource Type:

Published Article

Journal Name:

Monthly Notices of the Royal Astronomical Society

Additional Journal Information:

Journal Name: Monthly Notices of the Royal Astronomical Society Journal Volume: 510 Journal Issue: 2; Journal ID: ISSN 0035-8711

Publisher:

Oxford University Press

Country of Publication:

United Kingdom

Language:

English

Subject:

79 ASTRONOMY AND ASTROPHYSICS; Astronomy; Astrophysics; Data analysis; Large-scale structure of Universe