skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Nearest neighbour distributions: New statistical measures for cosmological clustering

Abstract

ABSTRACT The use of summary statistics beyond the two-point correlation function to analyse the non-Gaussian clustering on small scales, and thereby, increasing the sensitivity to the underlying cosmological parameters, is an active field of research in cosmology. In this paper, we explore a set of new summary statistics – the k-Nearest Neighbour Cumulative Distribution Functions (kNN-CDF). This is the empirical cumulative distribution function of distances from a set of volume-filling, Poisson distributed random points to the k-nearest data points, and is sensitive to all connected N-point correlations in the data. The kNN-CDF can be used to measure counts in cell, void probability distributions, and higher N-point correlation functions, all using the same formalism exploiting fast searches with spatial tree data structures. We demonstrate how it can be computed efficiently from various data sets – both discrete points, and the generalization for continuous fields. We use data from a large suite of N-body simulations to explore the sensitivity of this new statistic to various cosmological parameters, compared to the two-point correlation function, while using the same range of scales. We demonstrate that the use of kNN-CDF improves the constraints on the cosmological parameters by more than a factor of 2 when applied to the clustering of dark matter in the range of scales between 10 and $$40\, h^{-1}\, {\rm Mpc}$$. We also show that relative improvement is even greater when applied on the same scales to the clustering of haloes in the simulations at a fixed number density, both in real space, as well as in redshift space. Since the kNN-CDF are sensitive to all higher order connected correlation functions in the data, the gains over traditional two-point analyses are expected to grow as progressively smaller scales are included in the analysis of cosmological data, provided the higher order correlation functions are sensitive to cosmology on the scales of interest.

Authors:
ORCiD logo [1];  [1]
  1. Kavli Institute for Particle Astrophysics and Cosmology, Stanford University, 452 Lomita Mall, Stanford, CA 94305, USA, Department of Physics, Stanford University, 382 Via Pueblo Mall, Stanford, CA 94305, USA, SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, CA 94025, USA
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES)
OSTI Identifier:
1755286
Alternate Identifier(s):
OSTI ID: 1767152
Grant/Contract Number:  
AC02-76SF00515
Resource Type:
Published Article
Journal Name:
Monthly Notices of the Royal Astronomical Society
Additional Journal Information:
Journal Name: Monthly Notices of the Royal Astronomical Society Journal Volume: 500 Journal Issue: 4; Journal ID: ISSN 0035-8711
Publisher:
Royal Astronomical Society
Country of Publication:
United Kingdom
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; cosmological parameters; large-scale structure of Universe

Citation Formats

Banerjee, Arka, and Abel, Tom. Nearest neighbour distributions: New statistical measures for cosmological clustering. United Kingdom: N. p., 2020. Web. https://doi.org/10.1093/mnras/staa3604.
Banerjee, Arka, & Abel, Tom. Nearest neighbour distributions: New statistical measures for cosmological clustering. United Kingdom. https://doi.org/10.1093/mnras/staa3604
Banerjee, Arka, and Abel, Tom. Fri . "Nearest neighbour distributions: New statistical measures for cosmological clustering". United Kingdom. https://doi.org/10.1093/mnras/staa3604.
@article{osti_1755286,
title = {Nearest neighbour distributions: New statistical measures for cosmological clustering},
author = {Banerjee, Arka and Abel, Tom},
abstractNote = {ABSTRACT The use of summary statistics beyond the two-point correlation function to analyse the non-Gaussian clustering on small scales, and thereby, increasing the sensitivity to the underlying cosmological parameters, is an active field of research in cosmology. In this paper, we explore a set of new summary statistics – the k-Nearest Neighbour Cumulative Distribution Functions (kNN-CDF). This is the empirical cumulative distribution function of distances from a set of volume-filling, Poisson distributed random points to the k-nearest data points, and is sensitive to all connected N-point correlations in the data. The kNN-CDF can be used to measure counts in cell, void probability distributions, and higher N-point correlation functions, all using the same formalism exploiting fast searches with spatial tree data structures. We demonstrate how it can be computed efficiently from various data sets – both discrete points, and the generalization for continuous fields. We use data from a large suite of N-body simulations to explore the sensitivity of this new statistic to various cosmological parameters, compared to the two-point correlation function, while using the same range of scales. We demonstrate that the use of kNN-CDF improves the constraints on the cosmological parameters by more than a factor of 2 when applied to the clustering of dark matter in the range of scales between 10 and $40\, h^{-1}\, {\rm Mpc}$. We also show that relative improvement is even greater when applied on the same scales to the clustering of haloes in the simulations at a fixed number density, both in real space, as well as in redshift space. Since the kNN-CDF are sensitive to all higher order connected correlation functions in the data, the gains over traditional two-point analyses are expected to grow as progressively smaller scales are included in the analysis of cosmological data, provided the higher order correlation functions are sensitive to cosmology on the scales of interest.},
doi = {10.1093/mnras/staa3604},
journal = {Monthly Notices of the Royal Astronomical Society},
number = 4,
volume = 500,
place = {United Kingdom},
year = {2020},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1093/mnras/staa3604

Save / Share:

Works referenced in this record:

Superclusters
journal, September 1983


Eulerian BAO reconstructions and N -point statistics
journal, December 2015


A Global Descriptor of Spatial Pattern Interaction in the Galaxy Distribution
journal, March 1999

  • Kerscher, Martin; Pons‐Borderia, Maria Jesus; Schmalzing, Jens
  • The Astrophysical Journal, Vol. 513, Issue 2
  • DOI: 10.1086/306902

Statistics of cosmic density profiles from perturbation theory
journal, November 2014


The Quijote Simulations
journal, August 2020

  • Villaescusa-Navarro, Francisco; Hahn, ChangHoon; Massara, Elena
  • The Astrophysical Journal Supplement Series, Vol. 250, Issue 1
  • DOI: 10.3847/1538-4365/ab9d82

On the Trispectrum as a Gaussian Test for Cosmology
journal, May 2001

  • Verde, Licia; Heavens, Alan F.
  • The Astrophysical Journal, Vol. 553, Issue 1
  • DOI: 10.1086/320656

An accurate halo model for fitting non-linear cosmological power spectra and baryonic feedback models
journal, October 2015

  • Mead, A. J.; Peacock, J. A.; Heymans, C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 454, Issue 2
  • DOI: 10.1093/mnras/stv2036

Sufficient observables for large-scale structure in galaxy surveys
journal, December 2013

  • Carron, J.; Szapudi, I.
  • Monthly Notices of the Royal Astronomical Society: Letters, Vol. 439, Issue 1
  • DOI: 10.1093/mnrasl/slt167

Scale-dependent bias induced by local non-Gaussianity: a comparison to N -body simulations
journal, June 2009

  • Desjacques, Vincent; Seljak, Uroš; Iliev, Ilian T.
  • Monthly Notices of the Royal Astronomical Society, Vol. 396, Issue 1
  • DOI: 10.1111/j.1365-2966.2009.14721.x

The effective field theory of cosmological large scale structures
journal, September 2012

  • Carrasco, John Joseph M.; Hertzberg, Mark P.; Senatore, Leonardo
  • Journal of High Energy Physics, Vol. 2012, Issue 9
  • DOI: 10.1007/JHEP09(2012)082

The statistics of peaks of Gaussian random fields
journal, May 1986

  • Bardeen, J. M.; Bond, J. R.; Kaiser, N.
  • The Astrophysical Journal, Vol. 304
  • DOI: 10.1086/164143

Dark Energy Survey year 1 results: Cosmological constraints from galaxy clustering and weak lensing
journal, August 2018


Towards optimal extraction of cosmological information from nonlinear data
journal, December 2017

  • Seljak, Uroš; Aslanyan, Grigor; Feng, Yu
  • Journal of Cosmology and Astroparticle Physics, Vol. 2017, Issue 12
  • DOI: 10.1088/1475-7516/2017/12/009

Weighing neutrinos with the halo environment
journal, June 2020

  • Banerjee, Arka; Castorina, Emanuele; Villaescusa-Navarro, Francisco
  • Journal of Cosmology and Astroparticle Physics, Vol. 2020, Issue 06
  • DOI: 10.1088/1475-7516/2020/06/032

On the Incompleteness of the Moment and Correlation Function Hierarchy as Probes of the Lognormal Field
journal, August 2011


Weak lensing peak statistics in the era of large scale cosmological surveys
journal, October 2018


Density distribution of the cosmological matter field
journal, September 2018

  • Klypin, Anatoly; Prada, Francisco; Betancort-Rijo, Juan
  • Monthly Notices of the Royal Astronomical Society, Vol. 481, Issue 4
  • DOI: 10.1093/mnras/sty2613

Statistical Distributions, Third Edition
journal, December 2000


Back in the saddle: large-deviation statistics of the cosmic log-density field
journal, May 2016

  • Uhlemann, C.; Codis, S.; Pichon, C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 460, Issue 2
  • DOI: 10.1093/mnras/stw1074

A statistical comparison of voids in the galaxy distribution and n-body simulations
journal, December 1984

  • Ryden, B. S.; Turner, E. L.
  • The Astrophysical Journal, Vol. 287
  • DOI: 10.1086/184398

Statistics of voids in hierarchical universes
journal, July 1986


'Holes' in the Zwicky catalogue
journal, August 1981


Emulating the CFHTLenS weak lensing data: Cosmological constraints from moments and Minkowski functionals
journal, May 2015


Voronoi volume function: a new probe of cosmology and galaxy evolution
journal, January 2020

  • Paranjape, Aseem; Alam, Shadab
  • Monthly Notices of the Royal Astronomical Society, Vol. 495, Issue 3
  • DOI: 10.1093/mnras/staa1379

Detection of baryon acoustic oscillation features in the large-scale three-point correlation function of SDSS BOSS DR12 CMASS galaxies
journal, March 2017

  • Slepian, Zachary; Eisenstein, Daniel J.; Brownstein, Joel R.
  • Monthly Notices of the Royal Astronomical Society, Vol. 469, Issue 2
  • DOI: 10.1093/mnras/stx488

Improving Cosmological Distance Measurements by Reconstruction of the Baryon Acoustic Peak
journal, August 2007

  • Eisenstein, Daniel J.; Seo, Hee‐Jong; Sirko, Edwin
  • The Astrophysical Journal, Vol. 664, Issue 2
  • DOI: 10.1086/518712

Cosmological constraints with weak-lensing peak counts and second-order statistics in a large-field survey
journal, March 2017


On building fast kd-Trees for Ray Tracing, and on doing that in O(N log N)
conference, September 2006


From weak lensing to non-Gaussianity via Minkowski functionals: Weak lensing, non-Gaussianity and MFs
journal, November 2011

  • Munshi, Dipak; van Waerbeke, Ludovic; Smidt, Joseph
  • Monthly Notices of the Royal Astronomical Society, Vol. 419, Issue 1
  • DOI: 10.1111/j.1365-2966.2011.19718.x

Imprints of primordial non-Gaussianities on large-scale structure: Scale-dependent bias and abundance of virialized objects
journal, June 2008


The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample
journal, March 2017

  • Alam, Shadab; Ata, Metin; Bailey, Stephen
  • Monthly Notices of the Royal Astronomical Society, Vol. 470, Issue 3
  • DOI: 10.1093/mnras/stx721

Cosmological parameters from the BOSS galaxy power spectrum
journal, May 2020

  • Ivanov, Mikhail M.; Simonović, Marko; Zaldarriaga, Matias
  • Journal of Cosmology and Astroparticle Physics, Vol. 2020, Issue 05
  • DOI: 10.1088/1475-7516/2020/05/042

Simulations and symmetries
journal, January 2020

  • Modi, Chirag; Chen, Shi-Fan; White, Martin
  • Monthly Notices of the Royal Astronomical Society, Vol. 492, Issue 4
  • DOI: 10.1093/mnras/staa251

Evolution of one-point distributions from Gaussian initial fluctuations
journal, January 1994

  • Kofman, Lev; Bertschinger, Edmund; Gelb, James M.
  • The Astrophysical Journal, Vol. 420
  • DOI: 10.1086/173541

How to optimally constrain galaxy assembly bias: supplement projected correlation functions with count-in-cells statistics
journal, June 2019

  • Wang, Kuan; Mao, Yao-Yuan; Zentner, Andrew R.
  • Monthly Notices of the Royal Astronomical Society, Vol. 488, Issue 3
  • DOI: 10.1093/mnras/stz1733

A marked correlation function for constraining modified gravity models
journal, November 2016


Simulating nonlinear cosmological structure formation with massive neutrinos
journal, November 2016


Lagrangian perturbation theory at one loop order: Successes, failures, and improvements
journal, January 2015


corrfunc – a suite of blazing fast correlation functions on the CPU
journal, November 2019

  • Sinha, Manodeep; Garrison, Lehman H.
  • Monthly Notices of the Royal Astronomical Society, Vol. 491, Issue 2
  • DOI: 10.1093/mnras/stz3157

Rejuvenating the Matter Power Spectrum. iii. the Cosmology Sensitivity of Gaussianized Power Spectra
journal, November 2011


Cosmic web dependence of galaxy clustering and quenching in SDSS
journal, December 2018

  • Alam, Shadab; Zu, Ying; Peacock, John A.
  • Monthly Notices of the Royal Astronomical Society, Vol. 483, Issue 4
  • DOI: 10.1093/mnras/sty3477

Higher order statistics of the galaxy distribution using generating functions
journal, May 1993

  • Szapudi, Istvan; Szalay, Alexander S.
  • The Astrophysical Journal, Vol. 408
  • DOI: 10.1086/172568

Constraining neutrino mass with the tomographic weak lensing bispectrum
journal, May 2019

  • Coulton, William R.; Liu, Jia; Madhavacheril, Mathew S.
  • Journal of Cosmology and Astroparticle Physics, Vol. 2019, Issue 05
  • DOI: 10.1088/1475-7516/2019/05/043

Euclid preparation: II. The EuclidEmulator – a tool to compute the cosmology dependence of the nonlinear matter power spectrum
journal, January 2019

  • Knabenhans, Mischa; Stadel, Joachim; Marelli, Stefano
  • Monthly Notices of the Royal Astronomical Society, Vol. 484, Issue 4
  • DOI: 10.1093/mnras/stz197

Reconstructing baryon oscillations: A Lagrangian theory perspective
journal, March 2009


Cosmology and the bispectrum
journal, July 2006


A lognormal model for the cosmological mass distribution
journal, January 1991

  • Coles, Peter; Jones, Bernard
  • Monthly Notices of the Royal Astronomical Society, Vol. 248, Issue 1
  • DOI: 10.1093/mnras/248.1.1

Neutrino mass without cosmic variance
journal, May 2016


Testing modified gravity using a marked correlation function
journal, May 2018

  • Armijo, Joaquín; Cai, Yan-Chuan; Padilla, Nelson
  • Monthly Notices of the Royal Astronomical Society, Vol. 478, Issue 3
  • DOI: 10.1093/mnras/sty1335

The hierarchy of correlation functions and its relation to other measures of galaxy clustering
journal, February 1979

  • White, Simon D. M.
  • Monthly Notices of the Royal Astronomical Society, Vol. 186, Issue 2
  • DOI: 10.1093/mnras/186.2.145

The cosmological analysis of the SDSS/BOSS data from the Effective Field Theory of Large-Scale Structure
journal, May 2020

  • d'Amico, Guido; Gleyzes, Jérôme; Kokron, Nickolas
  • Journal of Cosmology and Astroparticle Physics, Vol. 2020, Issue 05
  • DOI: 10.1088/1475-7516/2020/05/005

Direct and fast calculation of regularized cosmological power spectrum at two-loop order
journal, November 2012


First Detection of Scale-Dependent Linear Halo Bias in N -Body Simulations with Massive Neutrinos
journal, February 2019


The Mira-Titan Universe. II. Matter Power Spectrum Emulation
journal, September 2017

  • Lawrence, Earl; Heitmann, Katrin; Kwan, Juliana
  • The Astrophysical Journal, Vol. 847, Issue 1
  • DOI: 10.3847/1538-4357/aa86a9

Cosmological reconstruction from galaxy light: neural network based light-matter connection
journal, October 2018


Structure in the Three-Dimensional Galaxy Distribution. i. Methods and Example Results
journal, December 2010


Nonlinear Evolution of the Bispectrum of Cosmological Perturbations
journal, April 1998

  • Scoccimarro, Roman; Colombi, Stephane; Fry, J. N.
  • The Astrophysical Journal, Vol. 496, Issue 2
  • DOI: 10.1086/305399

Density split statistics: Cosmological constraints from counts and lensing in cells in DES Y1 and SDSS data
journal, July 2018


Table for Estimating the Goodness of Fit of Empirical Distributions
journal, June 1948


Density split statistics: Joint model of counts and lensing in cells
journal, July 2018


Cosmology with Minkowski functionals and moments of the weak lensing convergence field
journal, December 2013


Fisher for complements: extracting cosmology and neutrino mass from the counts-in-cells PDF
journal, May 2020

  • Uhlemann, Cora; Friedrich, Oliver; Villaescusa-Navarro, Francisco
  • Monthly Notices of the Royal Astronomical Society, Vol. 495, Issue 4
  • DOI: 10.1093/mnras/staa1155

Constraining M ν with the bispectrum. Part I. Breaking parameter degeneracies
journal, March 2020

  • Hahn, ChangHoon; Villaescusa-Navarro, Francisco; Castorina, Emanuele
  • Journal of Cosmology and Astroparticle Physics, Vol. 2020, Issue 03
  • DOI: 10.1088/1475-7516/2020/03/040

Planck 2018 results: IX. Constraints on primordial non-Gaussianity
journal, September 2020


Marked clustering statistics in f(R) gravity cosmologies
journal, July 2018

  • Hernández-Aguayo, César; Baugh, Carlton M.; Li, Baojiu
  • Monthly Notices of the Royal Astronomical Society, Vol. 479, Issue 4
  • DOI: 10.1093/mnras/sty1822

Planck 2018 results: VI. Cosmological parameters
journal, September 2020


Algorithm for the direct reconstruction of the dark matter correlation function from weak lensing and galaxy clustering
journal, March 2010


On Information and Sufficiency
journal, March 1951

  • Kullback, S.; Leibler, R. A.
  • The Annals of Mathematical Statistics, Vol. 22, Issue 1
  • DOI: 10.1214/aoms/1177729694

A 2 per cent distance to z = 0.35 by reconstructing baryon acoustic oscillations – I. Methods and application to the Sloan Digital Sky Survey : A 2 per cent distance to z = 0.35
journal, November 2012

  • Padmanabhan, Nikhil; Xu, Xiaoying; Eisenstein, Daniel J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 427, Issue 3
  • DOI: 10.1111/j.1365-2966.2012.21888.x

Efficient optimal reconstruction of linear fields and band-powers from cosmological data
journal, October 2019


Primordial Non-Gaussianities and Zero-Bias Tracers of the Large-Scale Structure
journal, September 2018


Probing Galaxy assembly bias in BOSS galaxies using void probabilities
journal, May 2019

  • Walsh, Kilian; Tinker, Jeremy
  • Monthly Notices of the Royal Astronomical Society, Vol. 488, Issue 1
  • DOI: 10.1093/mnras/stz1351

Weak lensing by galaxy troughs in DES Science Verification data
journal, November 2015

  • Gruen, D.; Friedrich, O.; Amara, A.
  • Monthly Notices of the Royal Astronomical Society, Vol. 455, Issue 3
  • DOI: 10.1093/mnras/stv2506

The impact of galaxy formation on satellite kinematics and redshift-space distortions
journal, January 2018

  • Orsi, Álvaro A.; Angulo, Raúl E.
  • Monthly Notices of the Royal Astronomical Society, Vol. 475, Issue 2
  • DOI: 10.1093/mnras/stx3349

Measuring Cosmological Parameters with Galaxy Surveys
journal, November 1997


The power spectrum and bispectrum of SDSS DR11 BOSS galaxies – I. Bias and gravity
journal, May 2015

  • Gil-Marín, Héctor; Noreña, Jorge; Verde, Licia
  • Monthly Notices of the Royal Astronomical Society, Vol. 451, Issue 1
  • DOI: 10.1093/mnras/stv961

Void statistics and hierarchical scaling in the halo model
journal, May 2013

  • Fry, J. N.; Colombi, S.
  • Monthly Notices of the Royal Astronomical Society, Vol. 433, Issue 1
  • DOI: 10.1093/mnras/stt745

KiDS-450: cosmological parameter constraints from tomographic weak gravitational lensing
journal, November 2016

  • Hildebrandt, H.; Viola, M.; Heymans, C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 465, Issue 2
  • DOI: 10.1093/mnras/stw2805

Cosmological parameters from lensing power spectrum and bispectrum tomography
journal, March 2004


Gravitational Evolution of the Large‐Scale Probability Density Distribution: The Edgeworth and Gamma Expansions
journal, August 2000

  • Gaztanaga, E.; Fosalba, P.; Elizalde, E.
  • The Astrophysical Journal, Vol. 539, Issue 2
  • DOI: 10.1086/309249

NINE-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE ( WMAP ) OBSERVATIONS: COSMOLOGICAL PARAMETER RESULTS
journal, September 2013

  • Hinshaw, G.; Larson, D.; Komatsu, E.
  • The Astrophysical Journal Supplement Series, Vol. 208, Issue 2
  • DOI: 10.1088/0067-0049/208/2/19

Ringing the initial Universe: the response of overdensity and transformed-density power spectra to initial spikes
journal, June 2013

  • Neyrinck, Mark C.; Yang, Lin Forrest
  • Monthly Notices of the Royal Astronomical Society, Vol. 433, Issue 2
  • DOI: 10.1093/mnras/stt949

Why your model parameter confidences might be too optimistic. Unbiased estimation of the inverse covariance matrix
journal, December 2006


Cosmology with massive neutrinos I: towards a realistic modeling of the relation between matter, haloes and galaxies
journal, March 2014

  • Villaescusa-Navarro, Francisco; Marulli, Federico; Viel, Matteo
  • Journal of Cosmology and Astroparticle Physics, Vol. 2014, Issue 03
  • DOI: 10.1088/1475-7516/2014/03/011

Galaxy bias and σ8 from counts in cells from the SDSS main sample
journal, August 2020

  • Repp, Andrew; Szapudi, István
  • Monthly Notices of the Royal Astronomical Society: Letters, Vol. 498, Issue 1
  • DOI: 10.1093/mnrasl/slaa139

Enhancing BOSS bispectrum cosmological constraints with maximal compression
journal, January 2019

  • Gualdi, Davide; Gil-Marín, Héctor; Schuhmann, Robert L.
  • Monthly Notices of the Royal Astronomical Society, Vol. 484, Issue 3
  • DOI: 10.1093/mnras/stz051

Moving around the cosmological parameter space: A nonlinear power spectrum reconstruction based on high-resolution cosmic responses
journal, December 2017