skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Approximating Photo- z PDFs for Large Surveys

Abstract

Modern galaxy surveys produce redshift probability density functions (PDFs) in addition to traditional photometric redshift (photo-z) point estimates. However, the storage of photo-z PDFs may present a challenge with increasingly large catalogs, as we face a trade-off between the accuracy of subsequent science measurements and the limitation of finite storage resources. This work presents qp, a Python package for manipulating parameterizations of one-dimensional PDFs, as suitable for photo-z PDF compression. We use qp to investigate the performance of three simple PDF storage formats (quantiles, samples, and step functions) as a function of the number of stored parameters on two realistic mock data sets, representative of upcoming surveys with different data qualities. We propose some best practices for choosing a photo-z PDF approximation scheme and demonstrate the approach on a science case using performance metrics on both ensembles of individual photo-z PDFs and an estimator of the overall redshift distribution function. We show that both the properties of the set of PDFs we wish to approximate and the fidelity metric(s) chosen affect the optimal parameterization. Additionally, we find that quantiles and samples outperform step functions, and we encourage further consideration of these formats for PDF approximation.

Authors:
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3]; ORCiD logo [4]; ORCiD logo [5]; ORCiD logo [6]
  1. New York Univ. (NYU), NY (United States). Center for Cosmology and Particle Physics and Dept. of Physics
  2. SLAC National Accelerator Lab., Menlo Park, CA (United States); Stanford Univ., CA (United States). Kavli Inst. for Particle Astrophysics and Cosmology
  3. Stanford Univ., CA (United States). Kavli Inst. for Particle Astrophysics and Cosmology and Dept. of Physics
  4. Univ. of Washington, Seattle, WA (United States). Dept. of Astronomy
  5. Univ. of California, Davis, CA (United States). Dept. of Physics
  6. Stanford Univ., CA (United States). Kavli Inst. for Particle Astrophysics and Cosmology
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States); Oak Ridge Inst. for Science and Education (ORISE), Oak Ridge, TN (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Org.:
USDOE Office of Science (SC), Workforce Development for Teachers and Scientists (WDTS) (SC-27); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22). Scientific User Facilities Division; National Science Foundation (NSF); National Inst. of Nuclear Physics and Particle Physics (IN2P3); Science and Technology Facilities Council (STFC); LSST Corp., Tucson, AZ (United States); National Center for Scientific Research (CNRS); E-infrastructure Leadership Council (ELC); GridPP Collaboration
Contributing Org.:
LSST Dark Energy Science Collaboration
OSTI Identifier:
1461824
Grant/Contract Number:  
AC02-76SF00515; AST-1517237; SC0014664; N56981CC; AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Astronomical Journal (Online)
Additional Journal Information:
Journal Name: Astronomical Journal (Online); Journal Volume: 156; Journal Issue: 1; Journal ID: ISSN 1538-3881
Publisher:
IOP Publishing - AAAS
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; 97 MATHEMATICS AND COMPUTING; astronomical databases: miscellaneous; catalogs; galaxies: distances and redshifts; methods: miscellaneous; methods: statistical

Citation Formats

Malz, A. I., Marshall, P. J., DeRose, J., Graham, M. L., Schmidt, S. J., and Wechsler, R. Approximating Photo-z PDFs for Large Surveys. United States: N. p., 2018. Web. doi:10.3847/1538-3881/aac6b5.
Malz, A. I., Marshall, P. J., DeRose, J., Graham, M. L., Schmidt, S. J., & Wechsler, R. Approximating Photo-z PDFs for Large Surveys. United States. doi:10.3847/1538-3881/aac6b5.
Malz, A. I., Marshall, P. J., DeRose, J., Graham, M. L., Schmidt, S. J., and Wechsler, R. Fri . "Approximating Photo-z PDFs for Large Surveys". United States. doi:10.3847/1538-3881/aac6b5. https://www.osti.gov/servlets/purl/1461824.
@article{osti_1461824,
title = {Approximating Photo-z PDFs for Large Surveys},
author = {Malz, A. I. and Marshall, P. J. and DeRose, J. and Graham, M. L. and Schmidt, S. J. and Wechsler, R.},
abstractNote = {Modern galaxy surveys produce redshift probability density functions (PDFs) in addition to traditional photometric redshift (photo-z) point estimates. However, the storage of photo-z PDFs may present a challenge with increasingly large catalogs, as we face a trade-off between the accuracy of subsequent science measurements and the limitation of finite storage resources. This work presents qp, a Python package for manipulating parameterizations of one-dimensional PDFs, as suitable for photo-z PDF compression. We use qp to investigate the performance of three simple PDF storage formats (quantiles, samples, and step functions) as a function of the number of stored parameters on two realistic mock data sets, representative of upcoming surveys with different data qualities. We propose some best practices for choosing a photo-z PDF approximation scheme and demonstrate the approach on a science case using performance metrics on both ensembles of individual photo-z PDFs and an estimator of the overall redshift distribution function. We show that both the properties of the set of PDFs we wish to approximate and the fidelity metric(s) chosen affect the optimal parameterization. Additionally, we find that quantiles and samples outperform step functions, and we encourage further consideration of these formats for PDF approximation.},
doi = {10.3847/1538-3881/aac6b5},
journal = {Astronomical Journal (Online)},
number = 1,
volume = 156,
place = {United States},
year = {2018},
month = {6}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

METAPHOR: Probability density estimation for machine learning based photometric redshifts
journal, October 2016

  • Amaro, V.; Cavuoti, S.; Brescia, M.
  • Proceedings of the International Astronomical Union, Vol. 12, Issue S325
  • DOI: 10.1017/S1743921317002186

Weighing the Giants – III. Methods and measurements of accurate galaxy cluster weak-lensing masses
journal, February 2014

  • Applegate, Douglas E.; von der Linden, Anja; Kelly, Patrick L.
  • Monthly Notices of the Royal Astronomical Society, Vol. 439, Issue 1
  • DOI: 10.1093/mnras/stt2129

The Gaia astrophysical parameters inference system (Apsis): Pre-launch description
journal, November 2013


Bayesian Photometric Redshift Estimation
journal, June 2000

  • Benitez, Narciso
  • The Astrophysical Journal, Vol. 536, Issue 2
  • DOI: 10.1086/308947

Redshift distributions of galaxies in the Dark Energy Survey Science Verification shear catalogue and implications for weak lensing
journal, August 2016


SOMz: photometric redshift PDFs with self-organizing maps and random atlas
journal, January 2014

  • Carrasco Kind, Matias; Brunner, Robert J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 438, Issue 4
  • DOI: 10.1093/mnras/stt2456

Sparse representation of photometric redshift probability density functions: preparing for petascale astronomy
journal, May 2014

  • Carrasco Kind, Matias; Brunner, Robert J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 441, Issue 4
  • DOI: 10.1093/mnras/stu827

METAPHOR: a machine-learning-based method for the probability density estimation of photometric redshifts
journal, November 2016

  • Cavuoti, S.; Amaro, V.; Brescia, M.
  • Monthly Notices of the Royal Astronomical Society, Vol. 465, Issue 2
  • DOI: 10.1093/mnras/stw2930

CFHTLenS and RCSLenS: testing photometric redshift distributions using angular cross-correlations with spectroscopic galaxy surveys
journal, September 2016

  • Choi, A.; Heymans, C.; Blake, C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 463, Issue 4
  • DOI: 10.1093/mnras/stw2241

Galaxy–galaxy lensing in the Dark Energy Survey Science Verification data
journal, November 2016

  • Clampitt, J.; Sánchez, C.; Kwan, J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 465, Issue 4
  • DOI: 10.1093/mnras/stw2988

An end-to-end simulation framework for the Large Synoptic Survey Telescope
conference, August 2014

  • Connolly, Andrew J.; Angeli, George Z.; Chandrasekharan, Srinivasan
  • SPIE Astronomical Telescopes + Instrumentation, SPIE Proceedings
  • DOI: 10.1117/12.2054953

The third data release of the Kilo-Degree Survey and associated data products
journal, August 2017


DNF – Galaxy photometric redshift by Directional Neighbourhood Fitting
journal, April 2016

  • De Vicente, J.; Sánchez, E.; Sevilla-Noarbe, I.
  • Monthly Notices of the Royal Astronomical Society, Vol. 459, Issue 3
  • DOI: 10.1093/mnras/stw857

The VIMOS VLT deep survey: First epoch VVDS-deep survey: 11 564 spectra with 17.5 
journal, August 2005


How sensitive are predicted galaxy luminosities to the choice of stellar population synthesis model?
journal, February 2014

  • Gonzalez-Perez, V.; Lacey, C. G.; Baugh, C. M.
  • Monthly Notices of the Royal Astronomical Society, Vol. 439, Issue 1
  • DOI: 10.1093/mnras/stt2410

Photometric Redshifts with the LSST: Evaluating Survey Observing Strategies
journal, January 2018

  • Graham, Melissa L.; Connolly, Andrew J.; Ivezić, Željko
  • The Astronomical Journal, Vol. 155, Issue 1
  • DOI: 10.3847/1538-3881/aa99d4

KiDS-450: tomographic cross-correlation of galaxy shear with Planck lensing
journal, July 2017

  • Harnois-Déraps, Joachim; Tröster, Tilman; Chisari, Nora Elisa
  • Monthly Notices of the Royal Astronomical Society, Vol. 471, Issue 2
  • DOI: 10.1093/mnras/stx1675

KiDS-450: cosmological parameter constraints from tomographic weak gravitational lensing
journal, November 2016

  • Hildebrandt, H.; Viola, M.; Heymans, C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 465, Issue 2
  • DOI: 10.1093/mnras/stw2805

Matplotlib: A 2D Graphics Environment
journal, January 2007


Accurate photometric redshifts for the CFHT legacy survey calibrated using the VIMOS VLT deep survey
journal, September 2006


The X-Ray Binary Population of the Nearby Dwarf Starburst Galaxy IC 10: Variable and Transient X-Ray Sources
journal, February 2017

  • Laycock, Silas; Cappallo, Rigel; Williams, Benjamin F.
  • The Astrophysical Journal, Vol. 836, Issue 1
  • DOI: 10.3847/1538-4357/836/1/50

Lightcone mock catalogues from semi-analytic models of galaxy formation – I. Construction and application to the BzK colour selection
journal, December 2012

  • Merson, Alexander I.; Baugh, Carlton M.; Helly, John C.
  • Monthly Notices of the Royal Astronomical Society, Vol. 429, Issue 1
  • DOI: 10.1093/mnras/sts355

Results from DROXO: IV. EXTraS discovery of an X-ray flare from the Class I protostar candidate ISO-Oph 85
journal, February 2016


Searching for galaxy clusters in the Kilo-Degree Survey
journal, February 2017


ANNz2: Photometric Redshift and Probability Distribution Function Estimation using Machine Learning
journal, August 2016


Photometric Redshift Probability Distributions for Galaxies in the sdss dr8
journal, August 2012

  • Sheldon, Erin S.; Cunha, Carlos E.; Mandelbaum, Rachel
  • The Astrophysical Journal Supplement Series, Vol. 201, Issue 2
  • DOI: 10.1088/0067-0049/201/2/32

Simulations of the formation, evolution and clustering of galaxies and quasars
journal, June 2005

  • Springel, Volker; White, Simon D. M.; Jenkins, Adrian
  • Nature, Vol. 435, Issue 7042
  • DOI: 10.1038/nature03597

The star Cluster Mass–Galactocentric Radius Relation: Implications for Cluster Formation
journal, December 2015


Photometric redshifts for Hyper Suprime-Cam Subaru Strategic Program Data Release 1
journal, October 2017

  • Tanaka, Masayuki; Coupon, Jean; Hsieh, Bau-Ching
  • Publications of the Astronomical Society of Japan, Vol. 70, Issue SP1
  • DOI: 10.1093/pasj/psx077

Cross-correlation of weak lensing and gamma rays: implications for the nature of dark matter
journal, February 2017

  • Tröster, Tilman; Camera, Stefano; Fornasa, Mattia
  • Monthly Notices of the Royal Astronomical Society, Vol. 467, Issue 3
  • DOI: 10.1093/mnras/stx365

The NumPy Array: A Structure for Efficient Numerical Computation
journal, March 2011

  • van der Walt, Stéfan; Colbert, S. Chris; Varoquaux, Gaël
  • Computing in Science & Engineering, Vol. 13, Issue 2
  • DOI: 10.1109/MCSE.2011.37

    Works referencing / citing this record:

    Galaxy formation and evolution science in the era of the Large Synoptic Survey Telescope
    journal, June 2019


    Nonparametric Star Formation History Reconstruction with Gaussian Processes. I. Counting Major Episodes of Star Formation
    journal, July 2019

    • Iyer, Kartheik G.; Gawiser, Eric; Faber, Sandra M.
    • The Astrophysical Journal, Vol. 879, Issue 2
    • DOI: 10.3847/1538-4357/ab2052