skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: An effective online data monitoring and saving strategy for large-scale climate simulations

Abstract

Large-scale climate simulation models have been developed and widely used to generate historical data and study future climate scenarios. These simulation models often have to run for a couple of months to understand the changes in the global climate over the course of decades. This long-duration simulation process creates a huge amount of data with both high temporal and spatial resolution information; however, how to effectively monitor and record the climate changes based on these large-scale simulation results that are continuously produced in real time still remains to be resolved. Due to the slow process of writing data to disk, the current practice is to save a snapshot of the simulation results at a constant, slow rate although the data generation process runs at a very high speed. This study proposes an effective online data monitoring and saving strategy over the temporal and spatial domains with the consideration of practical storage and memory capacity constraints. Finally, our proposed method is able to intelligently select and record the most informative extreme values in the raw data generated from real-time simulations in the context of better monitoring climate changes.

Authors:
 [1];  [2];  [2];  [1];  [3]
  1. Univ. of Wisconsin, Madison, WI (United States). Dept. of Industrial and Systems Engineering
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Climate Change Science Inst.
  3. Xi'an Jiaotong Univ., Xi'an (China). School of Management. State Key Lab. for Manufacturing Systems Engineering
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); National Science Foundation (NSF); National Natural Science Foundation of China (NNSFC)
OSTI Identifier:
1423034
Grant/Contract Number:  
AC05-00OR22725; CMMI-1362529; 71402133; 71602155; 71572138; 11501209
Resource Type:
Accepted Manuscript
Journal Name:
Quality Technology & Quantitative Management
Additional Journal Information:
Journal Volume: 16; Journal Issue: 3; Journal ID: ISSN 1684-3703
Publisher:
Taylor & Francis Online
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; 97 MATHEMATICS AND COMPUTING; big data; local extrema; raw simulation data; spatial and temporal domains

Citation Formats

Xian, Xiaochen, Archibald, Rick, Mayer, Benjamin, Liu, Kaibo, and Li, Jian. An effective online data monitoring and saving strategy for large-scale climate simulations. United States: N. p., 2018. Web. doi:10.1080/16843703.2017.1414112.
Xian, Xiaochen, Archibald, Rick, Mayer, Benjamin, Liu, Kaibo, & Li, Jian. An effective online data monitoring and saving strategy for large-scale climate simulations. United States. doi:10.1080/16843703.2017.1414112.
Xian, Xiaochen, Archibald, Rick, Mayer, Benjamin, Liu, Kaibo, and Li, Jian. Mon . "An effective online data monitoring and saving strategy for large-scale climate simulations". United States. doi:10.1080/16843703.2017.1414112. https://www.osti.gov/servlets/purl/1423034.
@article{osti_1423034,
title = {An effective online data monitoring and saving strategy for large-scale climate simulations},
author = {Xian, Xiaochen and Archibald, Rick and Mayer, Benjamin and Liu, Kaibo and Li, Jian},
abstractNote = {Large-scale climate simulation models have been developed and widely used to generate historical data and study future climate scenarios. These simulation models often have to run for a couple of months to understand the changes in the global climate over the course of decades. This long-duration simulation process creates a huge amount of data with both high temporal and spatial resolution information; however, how to effectively monitor and record the climate changes based on these large-scale simulation results that are continuously produced in real time still remains to be resolved. Due to the slow process of writing data to disk, the current practice is to save a snapshot of the simulation results at a constant, slow rate although the data generation process runs at a very high speed. This study proposes an effective online data monitoring and saving strategy over the temporal and spatial domains with the consideration of practical storage and memory capacity constraints. Finally, our proposed method is able to intelligently select and record the most informative extreme values in the raw data generated from real-time simulations in the context of better monitoring climate changes.},
doi = {10.1080/16843703.2017.1414112},
journal = {Quality Technology & Quantitative Management},
number = 3,
volume = 16,
place = {United States},
year = {2018},
month = {1}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 1 work
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Climate forecasting: Build high-resolution global climate models
journal, November 2014


A density-based statistical process control scheme for high-dimensional and mixed-type observations
journal, April 2012


A High-Order Discontinuous Galerkin Discretization with Multiwavelet-Based Grid Adaptation for Compressible Flows
journal, March 2014

  • Gerhard, Nils; Iacono, Francesca; May, Georg
  • Journal of Scientific Computing, Vol. 62, Issue 1
  • DOI: 10.1007/s10915-014-9846-9

High-dimensional process monitoring and change point detection using embedding distributions in reproducing kernel Hilbert space
journal, June 2014


Multiwavelet Discontinuous Galerkin-Accelerated Exact Linear Part (ELP) Method for the Shallow-Water Equations on the Cubed Sphere
journal, February 2011

  • Archibald, Rick; Evans, Katherine J.; Drake, John
  • Monthly Weather Review, Vol. 139, Issue 2
  • DOI: 10.1175/2010MWR3271.1

Increase of extreme events in a warming world
journal, October 2011

  • Rahmstorf, S.; Coumou, D.
  • Proceedings of the National Academy of Sciences, Vol. 108, Issue 44
  • DOI: 10.1073/pnas.1101766108

An Adaptive Sampling Strategy for Online High-Dimensional Process Monitoring
journal, August 2014


Compressing an Ensemble With Statistical Models: An Algorithm for Global 3D Spatio-Temporal Temperature
journal, July 2016


Progress in Fast, Accurate Multi-scale Climate Simulations
journal, January 2015


Data Reduction Analysis for Climate Data Sets
journal, October 2013

  • Liu, Songbin; Huang, Xiaomeng; Fu, Haohuan
  • International Journal of Parallel Programming, Vol. 43, Issue 3
  • DOI: 10.1007/s10766-013-0287-0

A standard test set for numerical approximations to the shallow water equations in spherical geometry
journal, September 1992