skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

This content will become publicly available on August 10, 2020

Title: Hierarchical Modeling and Statistical Calibration for Photometric Redshifts

Abstract

The cosmological exploitation of modern photometric galaxy surveys requires both accurate (unbiased) and precise (narrow) redshift probability distributions derived from broadband photometry. Existing methodologies do not meet these requirements. Standard template fitting delivers interpretable models and errors, but lacks the flexibility to learn inaccuracies in the observed photometry or spectral templates. Machine learning addresses those issues, but requires representative training data, and the resulting models and uncertainties cannot be interpreted in the context of a physical model or outside of the training data. We present a hierarchical modeling approach simultaneously addressing the issues of flexibility, interpretability, and generalization. It combines template fitting with flexible (machine-learning-like) models to correct the spectral templates, model their redshift distributions, and recalibrate the photometric observations. By optimizing the full posterior distribution of the model and solving for its (thousands of) parameters, one can perform a global statistical calibration of the data and the spectral energy distribution (SED) model. We apply this approach to the public Dark Energy Survey Science Verification data and show that it provides more accurate and compact redshift posterior distributions than existing methods, as well as insights into residual photometric and SED systematics. Here, the model is causal and makes predictions formore » future data (e.g., additional photometric bandpasses), and its internal parameters and components are interpretable. This approach does not formally require the training data to be complete or representative; in principle, it can even work in regimes in which few or no spectroscopic redshifts are available.« less

Authors:
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3]; ORCiD logo [4]
  1. New York Univ. (NYU), NY (United States). Dept. of Physics, Center for Cosmology and Particle Physics
  2. New York Univ. (NYU), NY (United States). Dept. of Physics, Center for Cosmology and Particle Physics, and Center for Data Science; Flatiron Inst., New York, NY (United States). Center for Computational Astrophysics; Max Planck Inst. fur Astronomie, Heidelberg (Germany)
  3. Stanford Univ., CA (United States). Kavli Inst. for Particle Astrophysics and Cosmology & Physics Dept.; SLAC National Accelerator Lab., Menlo Park, CA (United States)
  4. Stanford Univ., CA (United States). Kavli Inst. for Particle Astrophysics and Cosmology & Physics Dept.; Stanford Univ., CA (United States). Dept. of Physics
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1562502
Grant/Contract Number:  
AC02-76SF00515
Resource Type:
Accepted Manuscript
Journal Name:
The Astrophysical Journal (Online)
Additional Journal Information:
Journal Name: The Astrophysical Journal (Online); Journal Volume: 881; Journal Issue: 1; Journal ID: ISSN 1538-4357
Publisher:
Institute of Physics (IOP)
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; distances and redshifts; galaxies: photometry; galaxies: statistics; large-scale structure of universe

Citation Formats

Leistedt, Boris, Hogg, David W., Wechsler, Risa H., and DeRose, Joe. Hierarchical Modeling and Statistical Calibration for Photometric Redshifts. United States: N. p., 2019. Web. doi:10.3847/1538-4357/ab2d29.
Leistedt, Boris, Hogg, David W., Wechsler, Risa H., & DeRose, Joe. Hierarchical Modeling and Statistical Calibration for Photometric Redshifts. United States. doi:10.3847/1538-4357/ab2d29.
Leistedt, Boris, Hogg, David W., Wechsler, Risa H., and DeRose, Joe. Sat . "Hierarchical Modeling and Statistical Calibration for Photometric Redshifts". United States. doi:10.3847/1538-4357/ab2d29.
@article{osti_1562502,
title = {Hierarchical Modeling and Statistical Calibration for Photometric Redshifts},
author = {Leistedt, Boris and Hogg, David W. and Wechsler, Risa H. and DeRose, Joe},
abstractNote = {The cosmological exploitation of modern photometric galaxy surveys requires both accurate (unbiased) and precise (narrow) redshift probability distributions derived from broadband photometry. Existing methodologies do not meet these requirements. Standard template fitting delivers interpretable models and errors, but lacks the flexibility to learn inaccuracies in the observed photometry or spectral templates. Machine learning addresses those issues, but requires representative training data, and the resulting models and uncertainties cannot be interpreted in the context of a physical model or outside of the training data. We present a hierarchical modeling approach simultaneously addressing the issues of flexibility, interpretability, and generalization. It combines template fitting with flexible (machine-learning-like) models to correct the spectral templates, model their redshift distributions, and recalibrate the photometric observations. By optimizing the full posterior distribution of the model and solving for its (thousands of) parameters, one can perform a global statistical calibration of the data and the spectral energy distribution (SED) model. We apply this approach to the public Dark Energy Survey Science Verification data and show that it provides more accurate and compact redshift posterior distributions than existing methods, as well as insights into residual photometric and SED systematics. Here, the model is causal and makes predictions for future data (e.g., additional photometric bandpasses), and its internal parameters and components are interpretable. This approach does not formally require the training data to be complete or representative; in principle, it can even work in regimes in which few or no spectroscopic redshifts are available.},
doi = {10.3847/1538-4357/ab2d29},
journal = {The Astrophysical Journal (Online)},
number = 1,
volume = 881,
place = {United States},
year = {2019},
month = {8}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on August 10, 2020
Publisher's Version of Record

Save / Share:

Works referenced in this record:

OzDES multifibre spectroscopy for the Dark Energy Survey: 3-yr results and first data release
journal, July 2017

  • Childress, M. J.; Lidman, C.; Davis, T. M.
  • Monthly Notices of the Royal Astronomical Society, Vol. 472, Issue 1
  • DOI: 10.1093/mnras/stx1872

THE zCOSMOS 10k-BRIGHT SPECTROSCOPIC SAMPLE
journal, September 2009

  • Lilly, Simon J.; Le Brun, Vincent; Maier, Christian
  • The Astrophysical Journal Supplement Series, Vol. 184, Issue 2
  • DOI: 10.1088/0067-0049/184/2/218

TPZ: photometric redshift PDFs and ancillary information by using prediction trees and random forests
journal, May 2013

  • Carrasco Kind, Matias; Brunner, Robert J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 432, Issue 2
  • DOI: 10.1093/mnras/stt574

CMB lensing tomography with the DES Science Verification galaxies
journal, January 2016

  • Giannantonio, T.; Fosalba, P.; Cawthon, R.
  • Monthly Notices of the Royal Astronomical Society, Vol. 456, Issue 3
  • DOI: 10.1093/mnras/stv2678

K -Corrections and Filter Transformations in the Ultraviolet, Optical, and Near-Infrared
journal, January 2007

  • Blanton, Michael R.; Roweis, Sam
  • The Astronomical Journal, Vol. 133, Issue 2
  • DOI: 10.1086/510127

Bayesian Photometric Redshift Estimation
journal, June 2000

  • Benitez, Narciso
  • The Astrophysical Journal, Vol. 536, Issue 2
  • DOI: 10.1086/308947

The GOODS-MUSIC sample: a multicolour catalog of near-IR selected galaxies in the GOODS-South field
journal, March 2006


Galaxy clustering, photometric redshifts and diagnosis of systematics in the DES Science Verification data
journal, December 2015

  • Crocce, M.; Carretero, J.; Bauer, A. H.
  • Monthly Notices of the Royal Astronomical Society, Vol. 455, Issue 4
  • DOI: 10.1093/mnras/stv2590

ANN z : Estimating Photometric Redshifts Using Artificial Neural Networks
journal, April 2004

  • Collister, Adrian A.; Lahav, Ofer
  • Publications of the Astronomical Society of the Pacific, Vol. 116, Issue 818
  • DOI: 10.1086/383254

Cosmology from cosmic shear with Dark Energy Survey Science Verification data
journal, July 2016


Reconstructing Redshift Distributions with Cross-Correlations: Tests and an Optimized Recipe
journal, August 2010


Cosmology from large-scale galaxy clustering and galaxy–galaxy lensing with Dark Energy Survey Science Verification data
journal, October 2016

  • Kwan, J.; Sánchez, C.; Clampitt, J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 464, Issue 4
  • DOI: 10.1093/mnras/stw2464

A Critical Assessment of Photometric Redshift Methods: a Candels Investigation
journal, September 2013


The DES Science Verification weak lensing shear catalogues
journal, May 2016

  • Jarvis, M.; Sheldon, E.; Zuntz, J.
  • Monthly Notices of the Royal Astronomical Society, Vol. 460, Issue 2
  • DOI: 10.1093/mnras/stw990

Cosmology constraints from shear peak statistics in Dark Energy Survey Science Verification data
journal, August 2016

  • Kacprzak, T.; Kirk, D.; Friedrich, O.
  • Monthly Notices of the Royal Astronomical Society, Vol. 463, Issue 4
  • DOI: 10.1093/mnras/stw2070

Mapping and Simulating Systematics due to Spatially Varying Observing Conditions in des Science Verification data
journal, October 2016

  • Leistedt, B.; Peiris, H. V.; Elsner, F.
  • The Astrophysical Journal Supplement Series, Vol. 226, Issue 2
  • DOI: 10.3847/0067-0049/226/2/24

The Arizona CDFS Environment Survey (ACES): A Magellan/IMACS Spectroscopic Survey of the Chandra Deep Field-South: Arizona CDFS Environment Survey (ACES)
journal, August 2012


Observational probes of cosmic acceleration
journal, September 2013


Colors and magnitudes predicted for high redshift galaxies
journal, July 1980

  • Coleman, G. D.; Wu, C. -C.; Weedman, D. W.
  • The Astrophysical Journal Supplement Series, Vol. 43
  • DOI: 10.1086/190674

Dark Energy Survey Year 1 Results: redshift distributions of the weak-lensing source galaxies
journal, April 2018

  • Hoyle, B.; Gruen, D.; Bernstein, G. M.
  • Monthly Notices of the Royal Astronomical Society, Vol. 478, Issue 1
  • DOI: 10.1093/mnras/sty957

The Kilo-Degree Survey
journal, August 2012

  • de Jong, Jelte T. A.; Verdoes Kleijn, Gijs A.; Kuijken, Konrad H.
  • Experimental Astronomy, Vol. 35, Issue 1-2
  • DOI: 10.1007/s10686-012-9306-1

Photometric redshift analysis in the Dark Energy Survey Science Verification data
journal, October 2014

  • Sánchez, C.; Carrasco Kind, M.; Lin, H.
  • Monthly Notices of the Royal Astronomical Society, Vol. 445, Issue 2
  • DOI: 10.1093/mnras/stu1836

Redshift distributions of galaxies in the Dark Energy Survey Science Verification shear catalogue and implications for weak lensing
journal, August 2016


Hierarchical Bayesian inference of galaxy redshift distributions from photometric surveys
journal, June 2016

  • Leistedt, Boris; Mortlock, Daniel J.; Peiris, Hiranya V.
  • Monthly Notices of the Royal Astronomical Society, Vol. 460, Issue 4
  • DOI: 10.1093/mnras/stw1304

Hierarchical Probabilistic Inference of the Color–Magnitude Diagram and Shrinkage of Stellar Distance Uncertainties
journal, November 2017


The Zurich Extragalactic Bayesian Redshift Analyzer and its first application: COSMOS
journal, October 2006


EAZY: A Fast, Public Photometric Redshift Code
journal, October 2008

  • Brammer, Gabriel B.; van Dokkum, Pieter G.; Coppi, Paolo
  • The Astrophysical Journal, Vol. 686, Issue 2
  • DOI: 10.1086/591786