Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Crowdsourcing quality control for Dark Energy Survey images

Abstract

We have developed here a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the “DES exposure checker”, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. Finally, we conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a su ciently large user base.

Authors:
ORCiD logo [1];  [2]; ORCiD logo [3];  [4];  [5];  [6];  [3];  [7];  [8];  [3];  [9];  [10];  [11];  [12];  [13];  [9];  [14];  [8];  [15];  [3] more »;  [3];  [16];  [12];  [15];  [17];  [18];  [1];  [5];  [19];  [20];  [21];  [9];  [19];  [21];  [3];  [9];  [22];  [23];  [24];  [3];  [25];  [5];  [3];  [19];  [26];  [15];  [27];  [5];  [3];  [15] « less
+ Show Author Affiliations
  1. The Ohio State Univ., Columbus, OH (United States)
  2. Brookhaven National Lab. (BNL), Upton, NY (United States)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  4. Stanford Univ., CA (United States). Kavli Inst.; SLAC National Accelerator Lab., Menlo Park, CA (United States)
  5. National Optical Astronomy Observatory, La Serena (Chile)
  6. Univ. College London, London (United Kingdom); Rhodes Univ., Grahamstown (South Aftrica)
  7. Centre National de la Recherche Scientifique (CNRS), Paris (France); Univ. College London, London (United Kingdom); Sorbonne Univ., Paris (France)
  8. Univ. College London, London (United Kingdom)
  9. Inter-Institutional e-Astronomy Lab (LIneA), Rio de Janeiro (Brazil); National Observatory, Rio de Janeiro (Brazil)
  10. Univ. of Illinois, Urbana, IL (United States); National Center for Supercomputing Applications, Urbana, IL (United States)
  11. Inst. of Space Sciences and Higher Council for Scientific Research, Barcelona (Spain); The Barcelona Institute of Science and Technology (BIST), Barcelona (Spain). Inst. of High Energy Physics
  12. Inst. of Space Sciences and Higher Council for Scientific Research, Barcelona (Spain)
  13. Univ. of Portsmouth (United Kingdom); Univ. of Southampton (United Kingdom)
  14. Excellence Cluster Universe, Garching (Germany); Ludwig-Maximilians Univ., Munich, (Germany)
  15. Univ. of Michigan, Ann Arbor, MI (United States)
  16. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Univ. of Chicago, IL (United States). Kavli Inst. for Cosmological Physics (KICP)
  17. Max Planck Inst. fur Extraterrestrische Physik, Garching (Germany); Univ. Sternwarte and Univ. Ludwig-Maximilians, Munchen (Germany)
  18. Univ. of Illinois, Urbana, IL (United States); National Center for Supercomputing Applications, Urbana, IL (United States);
  19. Univ. of Pennsylvania, Philadelphia, PA (United States)
  20. Australian Astronomical Observatory, North Ryde, NSW (Australia)
  21. Texas A&M Univ., College Station, TX (United States)
  22. California Inst. of Technology (CalTech), La Canada Flintridge, CA (United States). Jet Propulsion Lab.
  23. Univ. of Sussex, Brighton (United Kingdom)
  24. Center for Environmental and Technological Energy Research (CIEMAT), Madrid (Spain)
  25. Center for Environmental and Technological Energy Research (CIEMAT), Madrid (Spain); Univ. of Illinois, Urbana, IL (United States)
  26. National Center for Supercomputing Applications, Urbana, IL (United States)
  27. Argonne National Lab. (ANL), Argonne, IL (United States)
Publication Date:
Research Org.:
Brookhaven National Laboratory (BNL), Upton, NY (United States); Argonne National Laboratory (ANL), Argonne, IL (United States); SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); The Ohio State Univ., Columbus, OH (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP); National Science Foundation (NSF); USDOE Office of Science (SC), Basic Energy Sciences (BES); Science and Technology Facilities Council; University of Illinois-Urbana-Champaign - National Center for Supercomputing Applications; University of Chicago - Kavli Institute for Cosmological Physics
Contributing Org.:
DES
OSTI Identifier:
1336176
Alternate Identifier(s):
OSTI ID: 1275505; OSTI ID: 1346698; OSTI ID: 1356682; OSTI ID: 1602511
Report Number(s):
BNL-113170-2016-JA; arXiv:1511.03391; FERMILAB-PUB-15-487
Journal ID: ISSN 2213-1337; KA2301020
Grant/Contract Number:  
SC00112704; FG02-91ER40690; AC02-98CH10886; PHYS-1066293; AST-1138766; AYA2012-39559; ESP2013-48274; FPA2013-47986; SEV-2012-0234; AC02-07CH11359; AC02-06CH11357; SC0011726
Resource Type:
Accepted Manuscript
Journal Name:
Astronomy and Computing
Additional Journal Information:
Journal Volume: 16; Journal Issue: C; Journal ID: ISSN 2213-1337
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; Surveys; Information systems Crowdsourcing; Human-centered computing Collaborative filtering; 97 MATHEMATICS AND COMPUTING; Human-centered computing: Collaborative filtering; Information systems: Crowdsourcing; surveys; 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats

Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., and Zhang, Y. Crowdsourcing quality control for Dark Energy Survey images. United States: N. p., 2016. Web. doi:10.1016/j.ascom.2016.04.003.
Copy to clipboard
Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., & Zhang, Y. Crowdsourcing quality control for Dark Energy Survey images. United States. https://doi.org/10.1016/j.ascom.2016.04.003
Copy to clipboard
Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., and Zhang, Y. Wed . "Crowdsourcing quality control for Dark Energy Survey images". United States. https://doi.org/10.1016/j.ascom.2016.04.003. https://www.osti.gov/servlets/purl/1336176.
Copy to clipboard
@article{osti_1336176,
title = {Crowdsourcing quality control for Dark Energy Survey images},
author = {Melchior, P. and Sheldon, E. and Drlica-Wagner, A. and Rykoff, E. S. and Abbott, T. M. C. and Abdalla, F. B. and Allam, S. and Benoit-Lévy, A. and Brooks, D. and Buckley-Geer, E. and Carnero Rosell, A. and Carrasco Kind, M. and Carretero, J. and Crocce, M. and D’Andrea, C. B. and da Costa, L. N. and Desai, S. and Doel, P. and Evrard, A. E. and Finley, D. A. and Flaugher, B. and Frieman, J. and Gaztanaga, E. and Gerdes, D. W. and Gruen, D. and Gruendl, R. A. and Honscheid, K. and James, D. J. and Jarvis, M. and Kuehn, K. and Li, T. S. and Maia, M. A. G. and March, M. and Marshall, J. L. and Nord, B. and Ogando, R. and Plazas, A. A. and Romer, A. K. and Sanchez, E. and Scarpine, V. and Sevilla-Noarbe, I. and Smith, R. C. and Soares-Santos, M. and Suchyta, E. and Swanson, M. E. C. and Tarle, G. and Vikram, V. and Walker, A. R. and Wester, W. and Zhang, Y.},
abstractNote = {We have developed here a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the “DES exposure checker”, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. Finally, we conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a su ciently large user base.},
doi = {10.1016/j.ascom.2016.04.003},
journal = {Astronomy and Computing},
number = C,
volume = 16,
place = {United States},
year = {Wed May 11 00:00:00 EDT 2016},
month = {Wed May 11 00:00:00 EDT 2016}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1016/j.ascom.2016.04.003
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 9 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

SExtractor: Software for source extraction
journal, June 1996

  • Bertin, E.; Arnouts, S.
  • Astronomy and Astrophysics Supplement Series, Vol. 117, Issue 2
  • DOI: 10.1051/aas:1996164

The LSC glitch group: monitoring noise transients during the fifth LIGO science run
journal, September 2008

The Blanco Cosmology Survey: data Acquisition, Processing, Calibration, Quality Diagnostics, and data Release
journal, September 2012

Use of the Hough transformation to detect lines and curves in pictures
journal, January 1972

  • Duda, Richard O.; Hart, Peter E.
  • Communications of the ACM, Vol. 15, Issue 1
  • DOI: 10.1145/361237.361242

The dark Energy Camera
journal, October 2015

Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey ★
journal, September 2008

  • Lintott, Chris J.; Schawinski, Kevin; Slosar, Anže
  • Monthly Notices of the Royal Astronomical Society, Vol. 389, Issue 3
  • DOI: 10.1111/j.1365-2966.2008.13689.x

Crowdsourcing, the great meteor storm of 1833, and the founding of meteor science
journal, June 2014

A Modified Magnitude System that Produces Well-Behaved Magnitudes, Colors, and Errors Even for Low Signal-to-Noise Ratio Measurements
journal, September 1999

  • Lupton, Robert H.; Gunn, James E.; Szalay, Alexander S.
  • The Astronomical Journal, Vol. 118, Issue 3
  • DOI: 10.1086/301004

Space Warps – I. Crowdsourcing the discovery of gravitational lenses
journal, November 2015

  • Marshall, Philip J.; Verma, Aprajita; More, Anupreeta
  • Monthly Notices of the Royal Astronomical Society, Vol. 455, Issue 2
  • DOI: 10.1093/mnras/stv2009

A Review of Techniques for Extracting Linear Features from Imagery
journal, December 2004

The Sloan Digital Sky Survey: Technical Summary
journal, September 2000

  • York, Donald G.; Adelman, J.; Anderson, Jr., John E.
  • The Astronomical Journal, Vol. 120, Issue 3
  • DOI: 10.1086/301513
    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Dark Energy Survey Year 1 Results: The Photometric Data Set for Cosmology
    journal, April 2018

    • Drlica-Wagner, A.; Sevilla-Noarbe, I.; Rykoff, E. S.
    • The Astrophysical Journal Supplement Series, Vol. 235, Issue 2
    • DOI: 10.3847/1538-4365/aab4f5

    Comparative performance of selected variability detection techniques in photometric time series data
    journal, September 2016

    • Sokolovsky, K. V.; Gavras, P.; Karampelas, A.
    • Monthly Notices of the Royal Astronomical Society, Vol. 464, Issue 1
    • DOI: 10.1093/mnras/stw2262

    A transient search using combined human and machine classifications
    journal, July 2017

    • Wright, Darryl E.; Lintott, Chris J.; Smartt, Stephen J.
    • Monthly Notices of the Royal Astronomical Society, Vol. 472, Issue 2
    • DOI: 10.1093/mnras/stx1812

    MAXIMASK and MAXITRACK: Two new tools for identifying contaminants in astronomical images using convolutional neural networks
    journal, February 2020

    Variability search in M 31 using principal component analysis and the Hubble Source Catalogue
    journal, March 2018

    • Moretti, M. I.; Hatzidimitriou, D.; Karampelas, A.
    • Monthly Notices of the Royal Astronomical Society, Vol. 477, Issue 2
    • DOI: 10.1093/mnras/sty758

    Dark Energy Survey Year 1 Results: Photometric Data Set for Cosmology
    text, January 2018

    • Drlica-Wagner, A.; Sevilla-Noarbe, I.; Rykoff, Es
    • Apollo - University of Cambridge Repository
    • DOI: 10.17863/cam.21145

    Dark Energy Survey Year 1 Results: Photometric Data Set for Cosmology
    text, January 2017

      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: