The Scientific Data Management Center: Available Technologies and Highlights
Managing scientific data has been identified by the scientific community as one of the most important emerging needs because of the sheer volume and increasing complexity of data being collected. Effectively generating, managing, and analyzing this information requires a comprehensive, end-to-end approach to data management that encompasses all of the stages from the initial data acquisition to the final analysis of the data. Based on community input, we have identified three significant requirements. First, more efficient access to storage systems is needed. In particular, parallel file system and I/O system improvements are needed to write and read large volumes of data without slowing a simulation. Second, scientists require technologies to facilitate better understanding of their data, in particular the ability to effectively perform complex data analysis and searches over extremely large data sets. Furthermore, exploratory analysis requires techniques for efficiently selecting subsets of the data. Third, generating the data, collecting and storing the results, keeping track of data provenance, data post-processing, and analysis of results is a tedious, fragmented process. Tools for automation of this process in a robust, tractable, and recoverable fashion are required to enhance scientific exploration.
- Research Organization:
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-76RL01830
- OSTI ID:
- 1036433
- Report Number(s):
- PNNL-SA-80672; KJ0403000; TRN: US201206%%306
- Resource Relation:
- Conference: Proceedings of the Scientific Discovery through Advanced Computing Conference (SciDAC 2011), July 10-14, 2011, Denver, Colorado
- Country of Publication:
- United States
- Language:
- English
Similar Records
Neutron and X-ray Detectors
Data and Communications in Basic Energy Sciences: Creating a Pathway for Scientific Discovery