Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0)

Abstract

Abstract. Climate simulation codes, such as the Community Earth System Model (CESM), are especially complex and continually evolving. Their ongoing state of development requires frequent software verification in the form of quality assurance to both preserve the quality of the code and instill model confidence. To formalize and simplify this previously subjective and computationally expensive aspect of the verification process, we have developed a new tool for evaluating climate consistency. Because an ensemble of simulations allows us to gauge the natural variability of the model's climate, our new tool uses an ensemble approach for consistency testing. In particular, an ensemble of CESM climate runs is created, from which we obtain a statistical distribution that can be used to determine whether a new climate run is statistically distinguishable from the original ensemble. The CESM ensemble consistency test, referred to as CESM-ECT, is objective in nature and accessible to CESM developers and users. The tool has proven its utility in detecting errors in software and hardware environments and providing rapid feedback to model developers.

Authors:
 [1]; ORCiD logo [1];  [1];  [1];  [1]; ORCiD logo [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1];  [1]
+ Show Author Affiliations
  1. National Center for Atmospheric Research, Boulder, CO (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Univ. of California, Oakland, CA (United States); Argonne National Laboratory (ANL), Argonne, IL (United States); UT-Battelle LLC/ORNL, Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1565396
Grant/Contract Number:  
AC02-05CH11231; AC02-06CH11357; AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
Geoscientific Model Development (Online)
Additional Journal Information:
Journal Name: Geoscientific Model Development (Online); Journal Volume: 8; Journal Issue: 9; Journal ID: ISSN 1991-9603
Publisher:
European Geosciences Union
Country of Publication:
United States
Language:
English
Subject:
58 GEOSCIENCES; Geology

Citation Formats

Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D. A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0). United States: N. p., 2015. Web. doi:10.5194/gmd-8-2829-2015.
Copy to clipboard
Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., & Williamson, D. A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0). United States. https://doi.org/10.5194/gmd-8-2829-2015
Copy to clipboard
Baker, A. H., Hammerling, D. M., Levy, M. N., Xu, H., Dennis, J. M., Eaton, B. E., Edwards, J., Hannay, C., Mickelson, S. A., Neale, R. B., Nychka, D., Shollenberger, J., Tribbia, J., Vertenstein, M., and Williamson, D. Wed . "A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0)". United States. https://doi.org/10.5194/gmd-8-2829-2015. https://www.osti.gov/servlets/purl/1565396.
Copy to clipboard
@article{osti_1565396,
title = {A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0)},
author = {Baker, A. H. and Hammerling, D. M. and Levy, M. N. and Xu, H. and Dennis, J. M. and Eaton, B. E. and Edwards, J. and Hannay, C. and Mickelson, S. A. and Neale, R. B. and Nychka, D. and Shollenberger, J. and Tribbia, J. and Vertenstein, M. and Williamson, D.},
abstractNote = {Abstract. Climate simulation codes, such as the Community Earth System Model (CESM), are especially complex and continually evolving. Their ongoing state of development requires frequent software verification in the form of quality assurance to both preserve the quality of the code and instill model confidence. To formalize and simplify this previously subjective and computationally expensive aspect of the verification process, we have developed a new tool for evaluating climate consistency. Because an ensemble of simulations allows us to gauge the natural variability of the model's climate, our new tool uses an ensemble approach for consistency testing. In particular, an ensemble of CESM climate runs is created, from which we obtain a statistical distribution that can be used to determine whether a new climate run is statistically distinguishable from the original ensemble. The CESM ensemble consistency test, referred to as CESM-ECT, is objective in nature and accessible to CESM developers and users. The tool has proven its utility in detecting errors in software and hardware environments and providing rapid feedback to model developers.},
doi = {10.5194/gmd-8-2829-2015},
journal = {Geoscientific Model Development (Online)},
number = 9,
volume = 8,
place = {United States},
year = {Wed Sep 09 00:00:00 EDT 2015},
month = {Wed Sep 09 00:00:00 EDT 2015}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.5194/gmd-8-2829-2015
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 31 works
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: Percentage of variability explained for global mean by component scores.
All figures and tables (8 total)
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Ensemble Simulation of Twenty–First Century Climate Changes: Business–as–Usual versus CO 2 Stabilization
journal, November 2001

Guidelines for selecting and using simulation model verification techniques
conference, January 1989

  • Whitner, R. B.; Balci, O.
  • Proceedings of the 21st conference on Winter simulation - WSC '89
  • DOI: 10.1145/76738.76811

The Community Earth System Model: A Framework for Collaborative Research
journal, September 2013

  • Hurrell, James W.; Holland, M. M.; Gent, P. R.
  • Bulletin of the American Meteorological Society, Vol. 94, Issue 9
  • DOI: 10.1175/BAMS-D-12-00121.1

Engineering the Software for Understanding Climate Change
journal, November 2009

  • Easterbrook, Steve M.; Johns, Timothy C.
  • Computing in Science & Engineering, Vol. 11, Issue 6
  • DOI: 10.1109/MCSE.2009.193

Assessing climate model software quality: a defect density analysis of three models
journal, January 2012

Ensemble forecast: A new approach to uncertainty and predictability
journal, November 2005

Verification and validation of simulation models
conference, December 2011

  • Sargent, Robert G.
  • 2011 Winter Simulation Conference - (WSC 2011), Proceedings of the 2011 Winter Simulation Conference (WSC)
  • DOI: 10.1109/WSC.2011.6147750

Testing ensembles of climate change scenarios for “statistical significance”
journal, August 2012

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences
journal, February 1994

The Accumulation of Rounding Errors and Port Validation for Global Atmospheric Models
journal, March 1997

  • Rosinski, James M.; Williamson, David L.
  • SIAM Journal on Scientific Computing, Vol. 18, Issue 2
  • DOI: 10.1137/S1064827594275534

The Community Earth System Model (CESM) Large Ensemble Project: A Community Resource for Studying Climate Change in the Presence of Internal Climate Variability
journal, August 2015

  • Kay, J. E.; Deser, C.; Phillips, A.
  • Bulletin of the American Meteorological Society, Vol. 96, Issue 8
  • DOI: 10.1175/BAMS-D-13-00255.1

Evaluation (not validation) of quantitative models.
journal, December 1998

A methodology for evaluating the impact of data compression on climate simulation data
conference, January 2014

  • Baker, Allison H.; Xu, Haiying; Dennis, John M.
  • Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14
  • DOI: 10.1145/2600212.2600217

Guest Editors' Introduction: Climate Change - Science and Software
journal, November 2011

  • Easterbrook, Steve M.; Edwards, Paul N.; Balaji, Venkatramani
  • IEEE Software, Vol. 28, Issue 6
  • DOI: 10.1109/MS.2011.141

Simple Uncertainty Frameworks for Selecting Weighting Schemes and Interpreting Multimodel Ensemble Climate Change Experiments
journal, June 2013

  • Sansom, Philip G.; Stephenson, David B.; Ferro, Christopher A. T.
  • Journal of Climate, Vol. 26, Issue 12
  • DOI: 10.1175/JCLI-D-12-00462.1

Software Testing and Verification in Climate Model Development
journal, November 2011

The Paris Climate Agreement and future sea-level rise from Antarctica
journal, May 2021

Guidelines For Selecting And Using Simulation Model Verification Techniques
conference, January 1989

Verification and validation of simulation models
conference, January 1994

Verification and validation of simulation models
conference, December 2008

Verification and validation of simulation models
conference, December 2010

  • Sargent, Robert G.
  • 2010 Winter Simulation Conference - (WSC 2010), Proceedings of the 2010 Winter Simulation Conference
  • DOI: 10.1109/wsc.2010.5679166

Assessing climate model software quality: a defect density analysis of three models
posted_content, February 2012

Engineering the Software for Understanding Climate Change
journal, January 2009

Verification and Validation of Simulation Models
conference, January 2005

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Finding the Root Causes of Statistical Inconsistency in Community Earth System Model Output
    posted_content, February 2018

    Deploying RMACC Summit: An HPC Resource for the Rocky Mountain Region
    conference, January 2017

    • Anderson, Jonathon; Burns, Patrick J.; Milroy, Daniel
    • Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17
    • DOI: 10.1145/3093338.3093379

    A Multivariate Approach to Ensure Statistical Reproducibility of Climate Model Simulations
    conference, June 2019

    • Mahajan, Salil; Evans, Katherine J.; Kennedy, Joseph H.
    • PASC '19: Platform for Advanced Scientific Computing Conference, Proceedings of the Platform for Advanced Scientific Computing Conference
    • DOI: 10.1145/3324989.3325724

    Crossing the chasm: how to develop weather and climate models for next generation computers?
    journal, January 2018

    • Lawrence, Bryan N.; Rezny, Michael; Budich, Reinhard
    • Geoscientific Model Development, Vol. 11, Issue 5
    • DOI: 10.5194/gmd-11-1799-2018

    Nine time steps: ultra-fast statistical consistency testing of the Community Earth System Model (pyCECT v3.0)
    journal, April 2017

    • Milroy, Daniel J.; Baker, Allison H.; Hammerling, Dorit M.
    • Geoscientific Model Development Discussions
    • DOI: 10.5194/gmd-2017-49

    Crossing the chasm: how to develop weather and climate models for next generation computers?
    text, January 2018

    Nine time steps: ultra-fast statistical consistency testing of the Community Earth System Model (pyCECT v3.0)
    journal, January 2018

    • Milroy, Daniel J.; Baker, Allison H.; Hammerling, Dorit M.
    • Geoscientific Model Development, Vol. 11, Issue 2
    • DOI: 10.5194/gmd-11-697-2018
      Sort by:
      [ × clear filter / sort ]

      Figures / Tables found in this record:

      Figure 1 (p. 5)figure
      Figure 2 (p. 7)figure
      Table 1 (p. 8)table
      Table 2 (p. 8)table
      Figure 3 (p. 9)figure
      Figure 4 (p. 9)figure
      Table 3 (p. 10)table
      Table 4 (p. 11)table
        [ × clear filter / sort ]
        Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

        Similar Records in DOE PAGES and OSTI.GOV collections: