skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Galactos: computing the anisotropic 3-point correlation function for 2 billion galaxies

Abstract

The nature of dark energy and the complete theory of gravity are two central questions currently facing cosmology. A vital tool for addressing them is the 3-point correlation function (3PCF), which probes deviations from a spatially random distribution of galaxies. However, the 3PCF's formidable computational expense has prevented its application to astronomical surveys comprising millions to billions of galaxies. We present Galactos, a high-performance implementation of a novel, O( N 2) algorithm that uses a load-balanced k-d tree and spherical harmonic expansions to compute the anisotropic 3PCF. Our implementation is optimized for the Intel Xeon Phi architecture, exploiting SIMD parallelism, instruction and thread concurrency, and significant L1 and L2 cache reuse, reaching 39% of peak performance on a single node. Galactos scales to the full Cori system, achieving 9.8 PF (peak) and 5.06 PF (sustained) across 9636 nodes, making the 3PCF easily computable for all galaxies in the observable universe.

Authors:
 [1];  [2];  [1];  [2];  [1];  [2];  [1];  [2];  [1];  [3];  [1]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  2. Intel Corporation, Santa Clara, CA (United States)
  3. Harvard-Smithsonian Center for Astrophysics, Cambridge, MA (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1530305
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
SC '17 Proceedings
Additional Journal Information:
Conference: International Conference for High Performance Computing, Networking, Storage and Analysis, Denver, CO (United States), 12 - 17 NOV 2017
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; 97 MATHEMATICS AND COMPUTING

Citation Formats

Friesen, Brian, Dubey, Pradeep, Prabhat, ., Patwary, Md. Mostofa Ali, Austin, Brian, Satish, Nadathur, Slepian, Zachary, Sundaram, Narayanan, Bard, Deborah, Eisenstein, Daniel J., and Deslippe, Jack. Galactos: computing the anisotropic 3-point correlation function for 2 billion galaxies. United States: N. p., 2017. Web. doi:10.1145/3126908.3126927.
Friesen, Brian, Dubey, Pradeep, Prabhat, ., Patwary, Md. Mostofa Ali, Austin, Brian, Satish, Nadathur, Slepian, Zachary, Sundaram, Narayanan, Bard, Deborah, Eisenstein, Daniel J., & Deslippe, Jack. Galactos: computing the anisotropic 3-point correlation function for 2 billion galaxies. United States. doi:10.1145/3126908.3126927.
Friesen, Brian, Dubey, Pradeep, Prabhat, ., Patwary, Md. Mostofa Ali, Austin, Brian, Satish, Nadathur, Slepian, Zachary, Sundaram, Narayanan, Bard, Deborah, Eisenstein, Daniel J., and Deslippe, Jack. Sun . "Galactos: computing the anisotropic 3-point correlation function for 2 billion galaxies". United States. doi:10.1145/3126908.3126927. https://www.osti.gov/servlets/purl/1530305.
@article{osti_1530305,
title = {Galactos: computing the anisotropic 3-point correlation function for 2 billion galaxies},
author = {Friesen, Brian and Dubey, Pradeep and Prabhat, . and Patwary, Md. Mostofa Ali and Austin, Brian and Satish, Nadathur and Slepian, Zachary and Sundaram, Narayanan and Bard, Deborah and Eisenstein, Daniel J. and Deslippe, Jack},
abstractNote = {The nature of dark energy and the complete theory of gravity are two central questions currently facing cosmology. A vital tool for addressing them is the 3-point correlation function (3PCF), which probes deviations from a spatially random distribution of galaxies. However, the 3PCF's formidable computational expense has prevented its application to astronomical surveys comprising millions to billions of galaxies. We present Galactos, a high-performance implementation of a novel, O(N2) algorithm that uses a load-balanced k-d tree and spherical harmonic expansions to compute the anisotropic 3PCF. Our implementation is optimized for the Intel Xeon Phi architecture, exploiting SIMD parallelism, instruction and thread concurrency, and significant L1 and L2 cache reuse, reaching 39% of peak performance on a single node. Galactos scales to the full Cori system, achieving 9.8 PF (peak) and 5.06 PF (sustained) across 9636 nodes, making the 3PCF easily computable for all galaxies in the observable universe.},
doi = {10.1145/3126908.3126927},
journal = {SC '17 Proceedings},
number = ,
volume = ,
place = {United States},
year = {2017},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Figures / Tables:

Figure 1 Figure 1: Representation of a triangle configuration for the 3PCF. Each dot represents a galaxy. The anisotropic 3PCF depends on the vectors $\vec{r}$1 and $\vec{r}$2, the relative distances to the primary galaxy, which is at the bottom left vertex. The relevant quantities are the triangle side lengths r1 and r2,more » the angle between $\vec{r}$1 and $\vec{r}$2, and $\vec{r}$1 and $\vec{r}$2’s angles to the line of sight (dashed arrow). We expand the angular dependence of the anisotropic 3PCF in the basis of spherical harmonics, with the dependence on triangle side lengths r1 and r2 encoded in the radial coefficient ζ$^{m}_{ℓℓ′}$ (r1, r2). The panel on the right shows a schematic of the algorithm’s output: a coefficient ζ$^{m}_{ℓℓ′}$ as a function of the triangle side lengths r1 and r2. The color indicates the number of triangles; red is an excess over a spatially random distribution and blue a deficit. The features are from BAO.« less

Save / Share:

Works referenced in this record:

Dynamics of dark Energy
journal, November 2006

  • Copeland, Edmund J.; Sami, M.; Tsujikawa, Shinji
  • International Journal of Modern Physics D, Vol. 15, Issue 11
  • DOI: 10.1142/S021827180600942X

The skewness of the aperture mass statistic
journal, July 2004


Cosmic Complementarity: [ITAL]H[/ITAL][TINF]0[/TINF] and Ω[TINF][CLC][ITAL]m[/ITAL][/CLC][/TINF] from Combining Cosmic Microwave Background Experiments and Redshift Surveys
journal, September 1998

  • Eisenstein, Daniel J.; Hu, Wayne; Tegmark, Max
  • The Astrophysical Journal, Vol. 504, Issue 2
  • DOI: 10.1086/311582

The power spectrum and bispectrum of SDSS DR11 BOSS galaxies – I. Bias and gravity
journal, May 2015

  • Gil-Marín, Héctor; Noreña, Jorge; Verde, Licia
  • Monthly Notices of the Royal Astronomical Society, Vol. 451, Issue 1
  • DOI: 10.1093/mnras/stv961

The Bispectrum as a Signature of Gravitational Instability in Redshift Space
journal, June 1999

  • Scoccimarro, Roman; Couchman, H. M. P.; Frieman, Joshua A.
  • The Astrophysical Journal, Vol. 517, Issue 2
  • DOI: 10.1086/307220

The effect of large-scale structure on the SDSS galaxy three-point correlation function
journal, June 2006


Density Studies of mhd Interstellar Turbulence: Statistical Moments, Correlations and Bispectrum
journal, March 2009


Observational probes of cosmic acceleration
journal, September 2013


Three-Point Correlation Functions of sdss Galaxies: Constraining Galaxy-Mass bias
journal, September 2011

  • McBride, Cameron K.; Connolly, Andrew J.; Gardner, Jeffrey P.
  • The Astrophysical Journal, Vol. 739, Issue 2
  • DOI: 10.1088/0004-637X/739/2/85

Redshift distortions of galaxy correlation functions
journal, April 1994

  • Fry, J. N.; Gaztanaga, Enrique
  • The Astrophysical Journal, Vol. 425
  • DOI: 10.1086/173956

The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: RSD measurement from the power spectrum and bispectrum of the DR12 BOSS galaxies
journal, October 2016

  • Gil-Marín, Héctor; Percival, Will J.; Verde, Licia
  • Monthly Notices of the Royal Astronomical Society, Vol. 465, Issue 2
  • DOI: 10.1093/mnras/stw2679

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures
journal, January 2016


SEVEN-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE ( WMAP ) OBSERVATIONS: POWER SPECTRA AND WMAP -DERIVED PARAMETERS
journal, January 2011

  • Larson, D.; Dunkley, J.; Hinshaw, G.
  • The Astrophysical Journal Supplement Series, Vol. 192, Issue 2
  • DOI: 10.1088/0067-0049/192/2/16

Cosmic growth history and expansion history
journal, August 2005


Clustering of dark matter tracers: generalizing bias for the coming era of precision LSS
journal, August 2009


Lagrangian perturbations and the matter bispectrum II: the resummed one-loop correction to the matter bispectrum
journal, June 2012


Cosmology and the bispectrum
journal, July 2006


Computing the three-point correlation function of galaxies in $\mathcal {O}(N^2)$ time
journal, October 2015

  • Slepian, Zachary; Eisenstein, Daniel J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 454, Issue 4
  • DOI: 10.1093/mnras/stv2119

Modelling the large-scale redshift-space 3-point correlation function of galaxies
journal, March 2017

  • Slepian, Zachary; Eisenstein, Daniel J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 469, Issue 2
  • DOI: 10.1093/mnras/stx490

Constraining the baryon–dark matter relative velocity with the large-scale three-point correlation function of the SDSS BOSS DR12 CMASS galaxies
journal, October 2017

  • Slepian, Zachary; Eisenstein, Daniel J.; Blazek, Jonathan A.
  • Monthly Notices of the Royal Astronomical Society, Vol. 474, Issue 2
  • DOI: 10.1093/mnras/stx2723

Relative velocity of dark matter and baryonic fluids and the formation of the first structures
journal, October 2010


    Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.