Building a Large Scale Climate Data System in Support of HPC Environment
- ORNL
The Earth System Grid Federation (ESG) is a large scale, multi-institutional, interdisciplinary project that aims to provide climate scientists and impact policy makers worldwide a web-based and client-based platform to publish, disseminate, compare and analyze ever increasing climate related data. This paper describes our practical experiences on the design, development and operation of such a system. In particular, we focus on the support of the data lifecycle from a high performance computing (HPC) perspective that is critical to the end-to-end scientific discovery process. We discuss three subjects that interconnect the consumer and producer of scientific datasets: (1) the motivations, complexities and solutions of deep storage access and sharing in a tightly controlled environment; (2) the importance of scalable and flexible data publication/population; and (3) high performance indexing and search of data with geospatial properties. These perceived corner issues collectively contributed to the overall user experience and proved to be as important as any other architectural design considerations. Although the requirements and challenges are rooted and discussed from a climate science domain context, we believe the architectural problems, ideas and solutions discussed in this paper are generally useful and applicable in a larger scope.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). National Center for Computational Sciences (NCCS)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- DE-AC05-00OR22725
- OSTI ID:
- 1037642
- Resource Relation:
- Conference: The 7th International Conference on Next Generation Web Services Practices (NWeSP 2011), Salamanca, Spain, Spain, 20111019, 20111021
- Country of Publication:
- United States
- Language:
- English
Similar Records
DOE SciDAC's Earth System Grid Center for Enabling Technologies Final Report
Scalable Data-Intensive Geocomputation: A Design for Real-Time Continental Flood Inundation Mapping