skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The Dark Energy Survey Data Management System

Abstract

The Dark Energy Survey (DES) collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at the National Center for Supercomputing Applications (NCSA) and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes thatmore » are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used TeraGrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.« less

Authors:
; ; ; ; ; ; ; ; ; ;
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
938469
Report Number(s):
FERMILAB-CONF-08-313-CD
arXiv eprint number arXiv:0807.2515; TRN: US0901816
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Conference
Resource Relation:
Conference: To appear in the proceedings of SPIE Conference on Astronomical Telescopes and Instrumentation 2008, Marseille, France, 23-28 Jun 2008
Country of Publication:
United States
Language:
English
Subject:
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS, 99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; ACCELERATION; ASTRONOMY; FERMILAB; FLEXIBILITY; MAINTENANCE; MANAGEMENT; PERFORMANCE; PROCESSING; REPROCESSING; SHEAR; SKY; STORAGE; TELESCOPES; Astrophysics, Computing

Citation Formats

Mohr, Joseph J, /Illinois U., Urbana, Astron. Dept. /Illinois U., Urbana, Barkhouse, Wayne, /North Dakota U., Beldica, Cristina, /Illinois U., Urbana, Bertin, Emmanuel, /Paris, Inst. Astrophys., Dora Cai, Y, /NCSA, Urbana, Nicolaci da Costa, Luiz A, /Rio de Janeiro Observ., Darnell, J Anthony, /Illinois U., Urbana, Astron. Dept., Daues, Gregory E, /NCSA, Urbana, Jarvis, Michael, /Pennsylvania U., Gower, Michelle, /NCSA, Urbana, Lin, Huan, and /Fermilab /Rio de Janeiro Observ. The Dark Energy Survey Data Management System. United States: N. p., 2008. Web.
Mohr, Joseph J, /Illinois U., Urbana, Astron. Dept. /Illinois U., Urbana, Barkhouse, Wayne, /North Dakota U., Beldica, Cristina, /Illinois U., Urbana, Bertin, Emmanuel, /Paris, Inst. Astrophys., Dora Cai, Y, /NCSA, Urbana, Nicolaci da Costa, Luiz A, /Rio de Janeiro Observ., Darnell, J Anthony, /Illinois U., Urbana, Astron. Dept., Daues, Gregory E, /NCSA, Urbana, Jarvis, Michael, /Pennsylvania U., Gower, Michelle, /NCSA, Urbana, Lin, Huan, & /Fermilab /Rio de Janeiro Observ. The Dark Energy Survey Data Management System. United States.
Mohr, Joseph J, /Illinois U., Urbana, Astron. Dept. /Illinois U., Urbana, Barkhouse, Wayne, /North Dakota U., Beldica, Cristina, /Illinois U., Urbana, Bertin, Emmanuel, /Paris, Inst. Astrophys., Dora Cai, Y, /NCSA, Urbana, Nicolaci da Costa, Luiz A, /Rio de Janeiro Observ., Darnell, J Anthony, /Illinois U., Urbana, Astron. Dept., Daues, Gregory E, /NCSA, Urbana, Jarvis, Michael, /Pennsylvania U., Gower, Michelle, /NCSA, Urbana, Lin, Huan, and /Fermilab /Rio de Janeiro Observ. Tue . "The Dark Energy Survey Data Management System". United States. https://www.osti.gov/servlets/purl/938469.
@article{osti_938469,
title = {The Dark Energy Survey Data Management System},
author = {Mohr, Joseph J and /Illinois U., Urbana, Astron. Dept. /Illinois U., Urbana and Barkhouse, Wayne and /North Dakota U. and Beldica, Cristina and /Illinois U., Urbana and Bertin, Emmanuel and /Paris, Inst. Astrophys. and Dora Cai, Y and /NCSA, Urbana and Nicolaci da Costa, Luiz A and /Rio de Janeiro Observ. and Darnell, J Anthony and /Illinois U., Urbana, Astron. Dept. and Daues, Gregory E and /NCSA, Urbana and Jarvis, Michael and /Pennsylvania U. and Gower, Michelle and /NCSA, Urbana and Lin, Huan and /Fermilab /Rio de Janeiro Observ.},
abstractNote = {The Dark Energy Survey (DES) collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at the National Center for Supercomputing Applications (NCSA) and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used TeraGrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.},
doi = {},
url = {https://www.osti.gov/biblio/938469}, journal = {},
number = ,
volume = ,
place = {United States},
year = {2008},
month = {7}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: