skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity

Abstract

Evaluating a climate model’s fidelity (ability to simulate observed climate) is a critical step in establishing confidence in the model’s suitability for future climate projections, and in tuning climate model parameters. Model developers use their judgement in determining which trade-offs between different aspects of model fidelity are acceptable. However, little is known about the degree of consensus in these evaluations, and whether experts use the same criteria when different scientific objectives are defined. Here, we report on results from a broad community survey studying expert assessments of the relative importance of different output variables when evaluating a global atmospheric model’s mean climate. We find that experts adjust their ratings of variable importance in response to the scientific objective, for instance, scientists rate surface wind stress as significantly more important for Southern Ocean climate than for the water cycle in the Asian watershed. There is greater consensus on the importance of certain variables (e.g., shortwave cloud forcing) than others (e.g., aerosol optical depth). We find few differences in expert consensus between respondents with greater or less climate modeling experience, and no statistically significant differences between the responses of climate model developers and users. In conclusion, the concise variable lists and communitymore » ratings reported here provide baseline descriptive data on current expert understanding of certain aspects of model evaluation, and can serve as a starting point for further investigation, as well as developing more sophisticated evaluation and scoring criteria with respect to specific scientific objectives.« less

Authors:
 [1];  [1];  [1];  [1];  [1];  [1];  [1]
  1. Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1459662
Alternate Identifier(s):
OSTI ID: 1510431
Report Number(s):
PNNL-SA-129346
Journal ID: ISSN 0256-1530; PII: 7300
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Accepted Manuscript
Journal Name:
Advances in Atmospheric Sciences
Additional Journal Information:
Journal Volume: 35; Journal Issue: 9; Journal ID: ISSN 0256-1530
Publisher:
Springer
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; climate; climate model; model evaluation; numerical model skill; expert elicitation

Citation Formats

Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., and Qian, Yun. Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity. United States: N. p., 2018. Web. doi:10.1007/s00376-018-7300-x.
Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., & Qian, Yun. Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity. United States. doi:10.1007/s00376-018-7300-x.
Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., and Qian, Yun. Thu . "Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity". United States. doi:10.1007/s00376-018-7300-x. https://www.osti.gov/servlets/purl/1459662.
@article{osti_1459662,
title = {Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity},
author = {Burrows, Susannah M. and Dasgupta, Aritra and Reehl, Sarah and Bramer, Lisa and Ma, Po -Lun and Rasch, Philip J. and Qian, Yun},
abstractNote = {Evaluating a climate model’s fidelity (ability to simulate observed climate) is a critical step in establishing confidence in the model’s suitability for future climate projections, and in tuning climate model parameters. Model developers use their judgement in determining which trade-offs between different aspects of model fidelity are acceptable. However, little is known about the degree of consensus in these evaluations, and whether experts use the same criteria when different scientific objectives are defined. Here, we report on results from a broad community survey studying expert assessments of the relative importance of different output variables when evaluating a global atmospheric model’s mean climate. We find that experts adjust their ratings of variable importance in response to the scientific objective, for instance, scientists rate surface wind stress as significantly more important for Southern Ocean climate than for the water cycle in the Asian watershed. There is greater consensus on the importance of certain variables (e.g., shortwave cloud forcing) than others (e.g., aerosol optical depth). We find few differences in expert consensus between respondents with greater or less climate modeling experience, and no statistically significant differences between the responses of climate model developers and users. In conclusion, the concise variable lists and community ratings reported here provide baseline descriptive data on current expert understanding of certain aspects of model evaluation, and can serve as a starting point for further investigation, as well as developing more sophisticated evaluation and scoring criteria with respect to specific scientific objectives.},
doi = {10.1007/s00376-018-7300-x},
journal = {Advances in Atmospheric Sciences},
number = 9,
volume = 35,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Aerosols implicated as a prime driver of twentieth-century North Atlantic climate variability
journal, April 2012

  • Booth, Ben B. B.; Dunstone, Nick J.; Halloran, Paul R.
  • Nature, Vol. 484, Issue 7393
  • DOI: 10.1038/nature10946

Uncertainty Quantification in Climate Modeling and Projection
journal, May 2016

  • Qian, Yun; Jackson, Charles; Giorgi, Filippo
  • Bulletin of the American Meteorological Society, Vol. 97, Issue 5
  • DOI: 10.1175/BAMS-D-15-00297.1

Limitations of passive remote sensing to constrain global cloud condensation nuclei
journal, January 2016


A climate model projection weighting scheme accounting for performance and interdependence: Model Projection Weighting Scheme
journal, January 2017

  • Knutti, Reto; Sedláček, Jan; Sanderson, Benjamin M.
  • Geophysical Research Letters
  • DOI: 10.1002/2016GL072012

Improving our fundamental understanding of the role of aerosol−cloud interactions in the climate system
journal, May 2016

  • Seinfeld, John H.; Bretherton, Christopher; Carslaw, Kenneth S.
  • Proceedings of the National Academy of Sciences, Vol. 113, Issue 21
  • DOI: 10.1073/pnas.1514043113

Selecting a climate model subset to optimise key ensemble properties
journal, January 2018

  • Herger, Nadja; Abramowitz, Gab; Knutti, Reto
  • Earth System Dynamics, Vol. 9, Issue 1
  • DOI: 10.5194/esd-9-135-2018

Measuring Agreement about Ranked Decision Choices for a Single Subject
journal, January 2009

  • Riffenburgh, Robert H.; Johnstone, Peter A.
  • The International Journal of Biostatistics, Vol. 5, Issue 1
  • DOI: 10.2202/1557-4679.1113

Performance metrics for climate models
journal, January 2008

  • Gleckler, P. J.; Taylor, K. E.; Doutriaux, C.
  • Journal of Geophysical Research, Vol. 113, Issue D6
  • DOI: 10.1029/2007JD008972

Uncertain then, irrelevant now
journal, November 2013


How Well Do Coupled Models Simulate Today's Climate?
journal, March 2008

  • Reichler, Thomas; Kim, Junsu
  • Bulletin of the American Meteorological Society, Vol. 89, Issue 3
  • DOI: 10.1175/BAMS-89-3-303

Parametric sensitivity analysis of precipitation at global and local scales in the Community Atmosphere Model CAM5
journal, April 2015

  • Qian, Yun; Yan, Huiping; Hou, Zhangshuan
  • Journal of Advances in Modeling Earth Systems, Vol. 7, Issue 2
  • DOI: 10.1002/2014MS000354

The Art and Science of Climate Model Tuning
journal, March 2017

  • Hourdin, Frédéric; Mauritsen, Thorsten; Gettelman, Andrew
  • Bulletin of the American Meteorological Society, Vol. 98, Issue 3
  • DOI: 10.1175/BAMS-D-15-00135.1

A likelihood-based comparison of temporal models for physical processes
journal, April 2011

  • Braverman, Amy; Cressie, Noel; Teixeira, Joao
  • Statistical Analysis and Data Mining, Vol. 4, Issue 3
  • DOI: 10.1002/sam.10113

An automatic and effective parameter optimization method for model tuning
journal, January 2015


An Evaluation of Decadal Probability Forecasts from State-of-the-Art Climate Models
journal, December 2013


A new test statistic for climate models that includes field and spatial dependencies using Gaussian Markov random fields
journal, January 2016

  • Nosedal-Sanchez, Alvaro; Jackson, Charles S.; Huerta, Gabriel
  • Geoscientific Model Development, Vol. 9, Issue 7
  • DOI: 10.5194/gmd-9-2407-2016

    Works referencing / citing this record:

    A likelihood-based comparison of temporal models for physical processes
    journal, April 2011

    • Braverman, Amy; Cressie, Noel; Teixeira, Joao
    • Statistical Analysis and Data Mining, Vol. 4, Issue 3
    • DOI: 10.1002/sam.10113

    Uncertain then, irrelevant now
    journal, November 2013


    Aerosols implicated as a prime driver of twentieth-century North Atlantic climate variability
    journal, April 2012

    • Booth, Ben B. B.; Dunstone, Nick J.; Halloran, Paul R.
    • Nature, Vol. 484, Issue 7393
    • DOI: 10.1038/nature10946

    Improving our fundamental understanding of the role of aerosol−cloud interactions in the climate system
    journal, May 2016

    • Seinfeld, John H.; Bretherton, Christopher; Carslaw, Kenneth S.
    • Proceedings of the National Academy of Sciences, Vol. 113, Issue 21
    • DOI: 10.1073/pnas.1514043113

    Measuring Agreement about Ranked Decision Choices for a Single Subject
    journal, January 2009

    • Riffenburgh, Robert H.; Johnstone, Peter A.
    • The International Journal of Biostatistics, Vol. 5, Issue 1
    • DOI: 10.2202/1557-4679.1113

    Limitations of passive remote sensing to constrain global cloud condensation nuclei
    journal, January 2016


    Selecting a climate model subset to optimise key ensemble properties
    journal, January 2018

    • Herger, Nadja; Abramowitz, Gab; Knutti, Reto
    • Earth System Dynamics, Vol. 9, Issue 1
    • DOI: 10.5194/esd-9-135-2018

    An automatic and effective parameter optimization method for model tuning
    journal, January 2015


    A new test statistic for climate models that includes field and spatial dependencies using Gaussian Markov random fields
    journal, January 2016

    • Nosedal-Sanchez, Alvaro; Jackson, Charles S.; Huerta, Gabriel
    • Geoscientific Model Development, Vol. 9, Issue 7
    • DOI: 10.5194/gmd-9-2407-2016