skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Crowdsourcing quality control for Dark Energy Survey images

Abstract

We have developed here a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the “DES exposure checker”, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. Finally, we conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a su ciently large user base.

Authors:
ORCiD logo [1];  [2]; ORCiD logo [3];  [4];  [5];  [6];  [3];  [7];  [8];  [3];  [9];  [10];  [11];  [12];  [13];  [9];  [14];  [8];  [15];  [3] more »;  [3];  [16];  [12];  [15];  [17];  [18];  [1];  [5];  [19];  [20];  [21];  [9];  [19];  [21];  [3];  [9];  [22];  [23];  [24];  [3];  [25];  [5];  [3];  [19];  [26];  [15];  [27];  [5];  [3];  [15] « less
  1. The Ohio State Univ., Columbus, OH (United States)
  2. Brookhaven National Lab. (BNL), Upton, NY (United States)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  4. Stanford Univ., CA (United States). Kavli Inst.; SLAC National Accelerator Lab., Menlo Park, CA (United States)
  5. National Optical Astronomy Observatory, La Serena (Chile)
  6. Univ. College London, London (United Kingdom); Rhodes Univ., Grahamstown (South Aftrica)
  7. Centre National de la Recherche Scientifique (CNRS), Paris (France); Univ. College London, London (United Kingdom); Sorbonne Univ., Paris (France)
  8. Univ. College London, London (United Kingdom)
  9. Inter-Institutional e-Astronomy Lab (LIneA), Rio de Janeiro (Brazil); National Observatory, Rio de Janeiro (Brazil)
  10. Univ. of Illinois, Urbana, IL (United States); National Center for Supercomputing Applications, Urbana, IL (United States)
  11. Inst. of Space Sciences and Higher Council for Scientific Research, Barcelona (Spain); The Barcelona Institute of Science and Technology (BIST), Barcelona (Spain). Inst. of High Energy Physics
  12. Inst. of Space Sciences and Higher Council for Scientific Research, Barcelona (Spain)
  13. Univ. of Portsmouth (United Kingdom); Univ. of Southampton (United Kingdom)
  14. Excellence Cluster Universe, Garching (Germany); Ludwig-Maximilians Univ., Munich, (Germany)
  15. Univ. of Michigan, Ann Arbor, MI (United States)
  16. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Univ. of Chicago, IL (United States). Kavli Inst. for Cosmological Physics (KICP)
  17. Max Planck Inst. fur Extraterrestrische Physik, Garching (Germany); Univ. Sternwarte and Univ. Ludwig-Maximilians, Munchen (Germany)
  18. Univ. of Illinois, Urbana, IL (United States); National Center for Supercomputing Applications, Urbana, IL (United States);
  19. Univ. of Pennsylvania, Philadelphia, PA (United States)
  20. Australian Astronomical Observatory, North Ryde, NSW (Australia)
  21. Texas A&M Univ., College Station, TX (United States)
  22. California Inst. of Technology (CalTech), La Canada Flintridge, CA (United States). Jet Propulsion Lab.
  23. Univ. of Sussex, Brighton (United Kingdom)
  24. Center for Environmental and Technological Energy Research (CIEMAT), Madrid (Spain)
  25. Center for Environmental and Technological Energy Research (CIEMAT), Madrid (Spain); Univ. of Illinois, Urbana, IL (United States)
  26. National Center for Supercomputing Applications, Urbana, IL (United States)
  27. Argonne National Lab. (ANL), Argonne, IL (United States)
Publication Date:
Research Org.:
Brookhaven National Lab. (BNL), Upton, NY (United States); Argonne National Lab. (ANL), Argonne, IL (United States); SLAC National Accelerator Lab., Menlo Park, CA (United States); Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); The Ohio State Univ., Columbus, OH (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25); National Science Foundation (NSF); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22); Science and Technology Facilities Council; University of Illinois-Urbana-Champaign - National Center for Supercomputing Applications; University of Chicago - Kavli Institute for Cosmological Physics
Contributing Org.:
DES
OSTI Identifier:
1336176
Alternate Identifier(s):
OSTI ID: 1275505; OSTI ID: 1346698; OSTI ID: 1356682; OSTI ID: 1602511
Report Number(s):
BNL-113170-2016-JA; arXiv:1511.03391; FERMILAB-PUB-15-487
Journal ID: ISSN 2213-1337; KA2301020
Grant/Contract Number:  
SC00112704; FG02-91ER40690; AC02-98CH10886; PHYS-1066293; AST-1138766; AYA2012-39559; ESP2013-48274; FPA2013-47986; SEV-2012-0234; AC02-07CH11359; AC02-06CH11357; SC0011726
Resource Type:
Accepted Manuscript
Journal Name:
Astronomy and Computing
Additional Journal Information:
Journal Volume: 16; Journal Issue: C; Journal ID: ISSN 2213-1337
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; Surveys; Information systems Crowdsourcing; Human-centered computing Collaborative filtering; 97 MATHEMATICS AND COMPUTING; Human-centered computing: Collaborative filtering; Information systems: Crowdsourcing; surveys; 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats

Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., and Zhang, Y. Crowdsourcing quality control for Dark Energy Survey images. United States: N. p., 2016. Web. doi:10.1016/j.ascom.2016.04.003.
Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., & Zhang, Y. Crowdsourcing quality control for Dark Energy Survey images. United States. doi:10.1016/j.ascom.2016.04.003.
Melchior, P., Sheldon, E., Drlica-Wagner, A., Rykoff, E. S., Abbott, T. M. C., Abdalla, F. B., Allam, S., Benoit-Lévy, A., Brooks, D., Buckley-Geer, E., Carnero Rosell, A., Carrasco Kind, M., Carretero, J., Crocce, M., D’Andrea, C. B., da Costa, L. N., Desai, S., Doel, P., Evrard, A. E., Finley, D. A., Flaugher, B., Frieman, J., Gaztanaga, E., Gerdes, D. W., Gruen, D., Gruendl, R. A., Honscheid, K., James, D. J., Jarvis, M., Kuehn, K., Li, T. S., Maia, M. A. G., March, M., Marshall, J. L., Nord, B., Ogando, R., Plazas, A. A., Romer, A. K., Sanchez, E., Scarpine, V., Sevilla-Noarbe, I., Smith, R. C., Soares-Santos, M., Suchyta, E., Swanson, M. E. C., Tarle, G., Vikram, V., Walker, A. R., Wester, W., and Zhang, Y. Wed . "Crowdsourcing quality control for Dark Energy Survey images". United States. doi:10.1016/j.ascom.2016.04.003. https://www.osti.gov/servlets/purl/1336176.
@article{osti_1336176,
title = {Crowdsourcing quality control for Dark Energy Survey images},
author = {Melchior, P. and Sheldon, E. and Drlica-Wagner, A. and Rykoff, E. S. and Abbott, T. M. C. and Abdalla, F. B. and Allam, S. and Benoit-Lévy, A. and Brooks, D. and Buckley-Geer, E. and Carnero Rosell, A. and Carrasco Kind, M. and Carretero, J. and Crocce, M. and D’Andrea, C. B. and da Costa, L. N. and Desai, S. and Doel, P. and Evrard, A. E. and Finley, D. A. and Flaugher, B. and Frieman, J. and Gaztanaga, E. and Gerdes, D. W. and Gruen, D. and Gruendl, R. A. and Honscheid, K. and James, D. J. and Jarvis, M. and Kuehn, K. and Li, T. S. and Maia, M. A. G. and March, M. and Marshall, J. L. and Nord, B. and Ogando, R. and Plazas, A. A. and Romer, A. K. and Sanchez, E. and Scarpine, V. and Sevilla-Noarbe, I. and Smith, R. C. and Soares-Santos, M. and Suchyta, E. and Swanson, M. E. C. and Tarle, G. and Vikram, V. and Walker, A. R. and Wester, W. and Zhang, Y.},
abstractNote = {We have developed here a crowdsourcing web application for image quality control employed by the Dark Energy Survey. Dubbed the “DES exposure checker”, it renders science-grade images directly to a web browser and allows users to mark problematic features from a set of predefined classes. Users can also generate custom labels and thus help identify previously unknown problem classes. User reports are fed back to hardware and software experts to help mitigate and eliminate recognized issues. We report on the implementation of the application and our experience with its over 100 users, the majority of which are professional or prospective astronomers but not data management experts. We discuss aspects of user training and engagement, and demonstrate how problem reports have been pivotal to rapidly correct artifacts which would likely have been too subtle or infrequent to be recognized otherwise. Finally, we conclude with a number of important lessons learned, suggest possible improvements, and recommend this collective exploratory approach for future astronomical surveys or other extensive data sets with a su ciently large user base.},
doi = {10.1016/j.ascom.2016.04.003},
journal = {Astronomy and Computing},
number = C,
volume = 16,
place = {United States},
year = {2016},
month = {5}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share: