skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs

Abstract

The use of high-quality simulated sky catalogs is essential for the success of cosmological surveys. The catalogs have diverse applications, such as investigating signatures of fundamental physics in cosmological observables, understanding the effect of systematic uncertainties on measured signals and testing mitigation strategies for reducing these uncertainties, aiding analysis pipeline development and testing, and survey strategy optimization. The list of applications is growing with improvements in the quality of the catalogs and the details that they can provide. Given the importance of simulated catalogs, it is critical to provide rigorous validation protocols that enable both catalog providers and users to assess the quality of the catalogs in a straightforward and comprehensive way. For this purpose, we have developed the DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science Collaboration as well as for the broader community. The goal of DESCQA is to enable the inspection, validation, and comparison of an inhomogeneous set of synthetic catalogs via the provision of a common interface within an automated framework. In this paper, we present the design concept and first implementation of DESCQA. In order to establish and demonstrate its full functionality we use a set of interim catalogs and validation tests.more » We highlight several important aspects, both technical and scientific, that require thoughtful consideration when designing a validation framework, including validation metrics and how these metrics impose requirements on the synthetic sky catalogs.« less

Authors:
ORCiD logo; ; ORCiD logo; ; ORCiD logo; ; ; ; ; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo; ORCiD logo more »; ORCiD logo; ORCiD logo; ; ORCiD logo; ORCiD logo; ; ORCiD logo « less
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE
Contributing Org.:
The LSST Dark Energy Science Collaboration
OSTI Identifier:
1424028
Alternate Identifier(s):
OSTI ID: 1493251
Grant/Contract Number:  
AC02-76SF00515; AC02-05CH11231
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
The Astrophysical Journal. Supplement Series (Online)
Additional Journal Information:
Journal Volume: 234; Journal Issue: 2; Journal ID: ISSN 1538-4365
Publisher:
American Astronomical Society/IOP
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; large-scale structure of universe; methods: numerical

Citation Formats

Mao, Yao-Yuan, Kovacs, Eve, Heitmann, Katrin, Uram, Thomas D., Benson, Andrew J., Campbell, Duncan, Cora, Sofía A., DeRose, Joseph, Matteo, Tiziana Di, Habib, Salman, Hearin, Andrew P., Kalmbach, J. Bryce, Krughoff, K. Simon, Lanusse, François, Lukić, Zarija, Mandelbaum, Rachel, Newman, Jeffrey A., Padilla, Nelson, Paillas, Enrique, Pope, Adrian, Ricker, Paul M., Ruiz, Andrés N., Tenneti, Ananth, Vega-Martínez, Cristian A., Wechsler, Risa H., Zhou, Rongpu, and Zu, Ying. DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs. United States: N. p., 2018. Web. doi:10.3847/1538-4365/aaa6c3.
Mao, Yao-Yuan, Kovacs, Eve, Heitmann, Katrin, Uram, Thomas D., Benson, Andrew J., Campbell, Duncan, Cora, Sofía A., DeRose, Joseph, Matteo, Tiziana Di, Habib, Salman, Hearin, Andrew P., Kalmbach, J. Bryce, Krughoff, K. Simon, Lanusse, François, Lukić, Zarija, Mandelbaum, Rachel, Newman, Jeffrey A., Padilla, Nelson, Paillas, Enrique, Pope, Adrian, Ricker, Paul M., Ruiz, Andrés N., Tenneti, Ananth, Vega-Martínez, Cristian A., Wechsler, Risa H., Zhou, Rongpu, & Zu, Ying. DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs. United States. https://doi.org/10.3847/1538-4365/aaa6c3
Mao, Yao-Yuan, Kovacs, Eve, Heitmann, Katrin, Uram, Thomas D., Benson, Andrew J., Campbell, Duncan, Cora, Sofía A., DeRose, Joseph, Matteo, Tiziana Di, Habib, Salman, Hearin, Andrew P., Kalmbach, J. Bryce, Krughoff, K. Simon, Lanusse, François, Lukić, Zarija, Mandelbaum, Rachel, Newman, Jeffrey A., Padilla, Nelson, Paillas, Enrique, Pope, Adrian, Ricker, Paul M., Ruiz, Andrés N., Tenneti, Ananth, Vega-Martínez, Cristian A., Wechsler, Risa H., Zhou, Rongpu, and Zu, Ying. 2018. "DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs". United States. https://doi.org/10.3847/1538-4365/aaa6c3. https://www.osti.gov/servlets/purl/1424028.
@article{osti_1424028,
title = {DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs},
author = {Mao, Yao-Yuan and Kovacs, Eve and Heitmann, Katrin and Uram, Thomas D. and Benson, Andrew J. and Campbell, Duncan and Cora, Sofía A. and DeRose, Joseph and Matteo, Tiziana Di and Habib, Salman and Hearin, Andrew P. and Kalmbach, J. Bryce and Krughoff, K. Simon and Lanusse, François and Lukić, Zarija and Mandelbaum, Rachel and Newman, Jeffrey A. and Padilla, Nelson and Paillas, Enrique and Pope, Adrian and Ricker, Paul M. and Ruiz, Andrés N. and Tenneti, Ananth and Vega-Martínez, Cristian A. and Wechsler, Risa H. and Zhou, Rongpu and Zu, Ying},
abstractNote = {The use of high-quality simulated sky catalogs is essential for the success of cosmological surveys. The catalogs have diverse applications, such as investigating signatures of fundamental physics in cosmological observables, understanding the effect of systematic uncertainties on measured signals and testing mitigation strategies for reducing these uncertainties, aiding analysis pipeline development and testing, and survey strategy optimization. The list of applications is growing with improvements in the quality of the catalogs and the details that they can provide. Given the importance of simulated catalogs, it is critical to provide rigorous validation protocols that enable both catalog providers and users to assess the quality of the catalogs in a straightforward and comprehensive way. For this purpose, we have developed the DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science Collaboration as well as for the broader community. The goal of DESCQA is to enable the inspection, validation, and comparison of an inhomogeneous set of synthetic catalogs via the provision of a common interface within an automated framework. In this paper, we present the design concept and first implementation of DESCQA. In order to establish and demonstrate its full functionality we use a set of interim catalogs and validation tests. We highlight several important aspects, both technical and scientific, that require thoughtful consideration when designing a validation framework, including validation metrics and how these metrics impose requirements on the synthetic sky catalogs.},
doi = {10.3847/1538-4365/aaa6c3},
url = {https://www.osti.gov/biblio/1424028}, journal = {The Astrophysical Journal. Supplement Series (Online)},
issn = {1538-4365},
number = 2,
volume = 234,
place = {United States},
year = {Thu Feb 08 00:00:00 EST 2018},
month = {Thu Feb 08 00:00:00 EST 2018}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 13 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Chemoarchaeological downsizing in a hierarchical universe: impact of a top-heavy IGIMF
journal, December 2014


Galaxy shapes and alignments in the MassiveBlack-II hydrodynamic and dark matter-only simulations
journal, August 2015


Strong bimodality in the host halo mass of central galaxies from galaxy–galaxy lensing
journal, February 2016


Mapping stellar content to dark matter haloes – II. Halo mass is the main driver of galaxy quenching
journal, January 2016


Analytic model for galaxy and dark matter clustering
journal, October 2000


Semi-analytic modelling of galaxy formation: the local Universe
journal, December 1999


High-redshift galaxies in the Hubble Deep Field: colour selection and star formation history to z   4
journal, December 1996


The formation and evolution of galaxies within merging dark matter haloes*
journal, September 1993


Large-scale bias and the peak background split
journal, September 1999


Linking halo mass to galaxy luminosity
journal, September 2004


What Shapes the Luminosity Function of Galaxies?
journal, December 2003


An Improved Photometric Calibration of the Sloan Digital Sky Survey Imaging Data
journal, February 2008


The nature of galaxy bias and clustering
journal, February 2000


A recipe for galaxy formation
journal, December 1994


Planck 2013 results. XVI. Cosmological parameters
journal, October 2014


Modelling feedback from stars and black holes in galaxy mergers
journal, August 2005


Toward a Halo Mass Function for Precision Cosmology: The Limits of Universality
journal, December 2008


The Seventh Data Release of the Sloan Digital Sky Survey
text, January 2009


The Deep2 Galaxy Redshift Survey: Design, Observations, data Reduction, and Redshifts
journal, August 2013


What determines satellite galaxy disruption?: What determines satellite galaxy disruption?
journal, February 2010


The distribution of stellar mass in the low-redshift Universe
journal, October 2009


The MassiveBlack-II simulation: the evolution of haloes and galaxies to z ∼ 0
journal, April 2015


Galaxy and Mass Assembly (GAMA): survey diagnostics and core data release: GAMA
journal, March 2011


Galaxy formation in the Planck cosmology – I. Matching the observed evolution of star formation rates, colours and stellar masses
journal, June 2015


Coevolution (Or Not) of Supermassive Black Holes and Host Galaxies
journal, August 2013


Statistical Properties of X‐Ray Clusters: Analytic and Numerical Comparisons
journal, March 1998


Galaxy formation and large-scale bias
journal, April 1997


The high mass end of the stellar mass function: Dependence on stellar population models and agreement between fits to the light profile
journal, January 2017


Astropy: A community Python package for astronomy
journal, September 2013


Galaxy halo occupation at high redshift
journal, January 2002


MultiDark simulations: the story of dark matter halo concentrations and density profiles
journal, February 2016


Halo mass function: baryon impact, fitting formulae, and implications for cluster cosmology
journal, December 2015


The Concentration Dependence of the Galaxy–Halo Connection: Modeling Assembly bias with Abundance Matching
journal, December 2016


The Dark Side of the Halo Occupation Distribution
journal, July 2004


K -Corrections and Filter Transformations in the Ultraviolet, Optical, and Near-Infrared
journal, January 2007


Galaxy formation through hierarchical clustering
journal, September 1991


Cosmological Simulations with TreeSPH
journal, July 1996


The Galactic Disk Mass Budget. I. Stellar Mass Function and Density
journal, June 2001


SEVEN-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE ( WMAP ) OBSERVATIONS: COSMOLOGICAL INTERPRETATION
journal, January 2011


Populating a cluster of galaxies - I. Results at \fontshape{it}{z}=0
journal, December 2001


On the Distribution of the Two-Sample Cramer-von Mises Criterion
journal, September 1962


The dark side of galaxy colour: evidence from new SDSS measurements of galaxy clustering and lensing
journal, August 2014


Halo occupation numbers and galaxy bias
journal, November 2000


The impact of galaxy formation on the total mass, mass profile and abundance of haloes
journal, June 2014


Structure of superclusters and supercluster formation - III. Quantitative study of the Local Supercluster
journal, February 1984


The massive end of the luminosity and stellar mass functions: dependence on the fit to the light profile
journal, September 2013


Stellar masses and star formation histories for 10 5 galaxies from the Sloan Digital Sky Survey
journal, May 2003


Spatial Correlation Function and Pairwise Velocity Dispersion of Galaxies: Cold Dark Matter Models versus the Las Campanas Survey
journal, February 1998


The Seventh data Release of the Sloan Digital sky Survey
journal, May 2009


The non-parametric model for linking galaxy luminosity with halo/subhalo mass
journal, September 2006


Stellar population synthesis at the resolution of 2003
journal, October 2003


SUSSING MERGER TREES: the influence of the halo finder
journal, May 2014


The Halo Occupation Distribution: Toward an Empirical Determination of the Relation between Galaxies and Mass
journal, August 2002


Galacticus: A semi-analytic model of galaxy formation
journal, February 2012


Effects of AGN feedback on ΛCDM galaxies
journal, August 2008


Mapping stellar content to dark matter haloes using galaxy clustering and galaxy–galaxy lensing in the SDSS DR7
journal, October 2015


Theoretical Models of the Halo Occupation Distribution: Separating Central and Satellite Galaxies
journal, November 2005


The Halo Mass Function: High‐Redshift Evolution and Universality
journal, December 2007


Modeling Luminosity‐dependent Galaxy Clustering through Cosmic Time
journal, August 2006


New York University Value-Added Galaxy Catalog: A Galaxy Catalog Based on New Public Surveys
journal, June 2005


Cosmological smoothed particle hydrodynamics simulations: a hybrid multiphase model for star formation
journal, February 2003


Galaxy formation theory
journal, October 2010


The 13th Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey Mapping Nearby Galaxies at Apache Point Observatory
journal, December 2017


The evolution of large-scale structure in a universe dominated by cold dark matter
journal, May 1985


A primer on hierarchical galaxy formation: the semi-analytical approach
journal, November 2006


A COMPREHENSIVE ANALYSIS OF UNCERTAINTIES AFFECTING THE STELLAR MASS-HALO MASS RELATION FOR 0 < z < 4
journal, June 2010


The Rockstar Phase-Space Temporal halo Finder and the Velocity Offsets of Cluster Cores
journal, December 2012


Gravitationally Consistent halo Catalogs and Merger Trees for Precision Cosmology
journal, December 2012


Mass Function Predictions Beyond Λcdm
journal, April 2011


Cold Flows and the First Quasars
journal, January 2012


Hierarchical Galaxy Growth and Scatter in the Stellar Mass–Halo mass Relation
journal, December 2016


Matplotlib: A 2D Graphics Environment
journal, January 2007


A Theoretical Framework for Combining Techniques that Probe the link Between Galaxies and dark Matter
journal, August 2011


Revisiting the Scaling Relations of Black hole Masses and host Galaxy Properties
journal, February 2013


A Critical Assessment of Stellar mass Measurement Methods
journal, July 2015


Constraints on the Relationship Between Stellar mass and halo mass at low and high Redshift
journal, January 2010


The Connection Between Galaxies and dark Matter Structures in the Local Universe
journal, June 2013


Calibration of Semi-Analytic Models of Galaxy Formation Using Particle Swarm Optimization
journal, March 2015


The NumPy Array: A Structure for Efficient Numerical Computation
journal, March 2011


CASSPER is a semantic segmentation-based particle picking algorithm for single-particle cryo-electron microscopy
journal, February 2021


Galaxy shapes and alignments in the MassiveBlack-II hydrodynamic and dark matter-only simulations
text, January 2015


The 13th Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey Mapping Nearby Galaxies at Apache Point Observatory
text, January 2017


Galaxy shapes and alignments in the MassiveBlack-II hydrodynamic and dark matter-only simulations
text, January 2015


The seventh data release of the sloan digital sky survey
text, January 2009


Effects of AGN feedback on LCDM galaxies
text, January 2008


Constraints on the relationship between stellar mass and halo mass at low and high redshift
text, January 2009


Galaxy Formation Theory
text, January 2010


The Connection between Galaxies and Dark Matter Structures in the Local Universe
text, January 2012


Planck 2013 results. XVI. Cosmological parameters
text, January 2013


The MassiveBlack-II Simulation: The Evolution of Halos and Galaxies to z~0
text, January 2014


Sussing Merger Trees: the influence of the halo finder
text, January 2014


Halo occupation numbers and galaxy bias
text, January 2000


Galaxy Halo Occupation at High Redshift
text, January 2001


What Shapes the Luminosity Function of Galaxies?
text, January 2003


The Dark Side of the Halo Occupation Distribution
text, January 2003


The Halo Mass Function: High-Redshift Evolution and Universality
text, January 2007


The Nature of Galaxy Bias and Clustering
text, January 1999


Works referencing / citing this record:

A Compendium of Extinction Curves for Simple Galactic Geometries
journal, October 2018


Mapping stellar content to dark matter haloes – III. Environmental dependence and conformity of galaxy colours
journal, February 2018


Covariances of galaxy stellar mass functions and correlation functions
journal, October 2018


CosmoDC2: A Synthetic Sky Catalog for Dark Energy Science with LSST
journal, December 2019


CosmoDC2: A Synthetic Sky Catalog for Dark Energy Science with LSST
text, January 2019