Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity
Abstract
Evaluating a climate model’s fidelity (ability to simulate observed climate) is a critical step in establishing confidence in the model’s suitability for future climate projections, and in tuning climate model parameters. Model developers use their judgement in determining which trade-offs between different aspects of model fidelity are acceptable. However, little is known about the degree of consensus in these evaluations, and whether experts use the same criteria when different scientific objectives are defined. Here, we report on results from a broad community survey studying expert assessments of the relative importance of different output variables when evaluating a global atmospheric model’s mean climate. We find that experts adjust their ratings of variable importance in response to the scientific objective, for instance, scientists rate surface wind stress as significantly more important for Southern Ocean climate than for the water cycle in the Asian watershed. There is greater consensus on the importance of certain variables (e.g., shortwave cloud forcing) than others (e.g., aerosol optical depth). We find few differences in expert consensus between respondents with greater or less climate modeling experience, and no statistically significant differences between the responses of climate model developers and users. In conclusion, the concise variable lists and communitymore »
- Authors:
-
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Publication Date:
- Research Org.:
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1459662
- Alternate Identifier(s):
- OSTI ID: 1510431
- Report Number(s):
- PNNL-SA-129346
Journal ID: ISSN 0256-1530; PII: 7300
- Grant/Contract Number:
- AC05-76RL01830
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Advances in Atmospheric Sciences
- Additional Journal Information:
- Journal Volume: 35; Journal Issue: 9; Journal ID: ISSN 0256-1530
- Publisher:
- Springer
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 54 ENVIRONMENTAL SCIENCES; climate; climate model; model evaluation; numerical model skill; expert elicitation
Citation Formats
Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., and Qian, Yun. Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity. United States: N. p., 2018.
Web. doi:10.1007/s00376-018-7300-x.
Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., & Qian, Yun. Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity. United States. doi:https://doi.org/10.1007/s00376-018-7300-x
Burrows, Susannah M., Dasgupta, Aritra, Reehl, Sarah, Bramer, Lisa, Ma, Po -Lun, Rasch, Philip J., and Qian, Yun. Thu .
"Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity". United States. doi:https://doi.org/10.1007/s00376-018-7300-x. https://www.osti.gov/servlets/purl/1459662.
@article{osti_1459662,
title = {Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity},
author = {Burrows, Susannah M. and Dasgupta, Aritra and Reehl, Sarah and Bramer, Lisa and Ma, Po -Lun and Rasch, Philip J. and Qian, Yun},
abstractNote = {Evaluating a climate model’s fidelity (ability to simulate observed climate) is a critical step in establishing confidence in the model’s suitability for future climate projections, and in tuning climate model parameters. Model developers use their judgement in determining which trade-offs between different aspects of model fidelity are acceptable. However, little is known about the degree of consensus in these evaluations, and whether experts use the same criteria when different scientific objectives are defined. Here, we report on results from a broad community survey studying expert assessments of the relative importance of different output variables when evaluating a global atmospheric model’s mean climate. We find that experts adjust their ratings of variable importance in response to the scientific objective, for instance, scientists rate surface wind stress as significantly more important for Southern Ocean climate than for the water cycle in the Asian watershed. There is greater consensus on the importance of certain variables (e.g., shortwave cloud forcing) than others (e.g., aerosol optical depth). We find few differences in expert consensus between respondents with greater or less climate modeling experience, and no statistically significant differences between the responses of climate model developers and users. In conclusion, the concise variable lists and community ratings reported here provide baseline descriptive data on current expert understanding of certain aspects of model evaluation, and can serve as a starting point for further investigation, as well as developing more sophisticated evaluation and scoring criteria with respect to specific scientific objectives.},
doi = {10.1007/s00376-018-7300-x},
journal = {Advances in Atmospheric Sciences},
number = 9,
volume = 35,
place = {United States},
year = {2018},
month = {7}
}
Works referenced in this record:
Aerosols implicated as a prime driver of twentieth-century North Atlantic climate variability
journal, April 2012
- Booth, Ben B. B.; Dunstone, Nick J.; Halloran, Paul R.
- Nature, Vol. 484, Issue 7393
Uncertainty Quantification in Climate Modeling and Projection
journal, May 2016
- Qian, Yun; Jackson, Charles; Giorgi, Filippo
- Bulletin of the American Meteorological Society, Vol. 97, Issue 5
Limitations of passive remote sensing to constrain global cloud condensation nuclei
journal, January 2016
- Stier, Philip
- Atmospheric Chemistry and Physics, Vol. 16, Issue 10
A climate model projection weighting scheme accounting for performance and interdependence: Model Projection Weighting Scheme
journal, January 2017
- Knutti, Reto; Sedláček, Jan; Sanderson, Benjamin M.
- Geophysical Research Letters
Improving our fundamental understanding of the role of aerosol−cloud interactions in the climate system
journal, May 2016
- Seinfeld, John H.; Bretherton, Christopher; Carslaw, Kenneth S.
- Proceedings of the National Academy of Sciences, Vol. 113, Issue 21
Selecting a climate model subset to optimise key ensemble properties
journal, January 2018
- Herger, Nadja; Abramowitz, Gab; Knutti, Reto
- Earth System Dynamics, Vol. 9, Issue 1
Measuring Agreement about Ranked Decision Choices for a Single Subject
journal, January 2009
- Riffenburgh, Robert H.; Johnstone, Peter A.
- The International Journal of Biostatistics, Vol. 5, Issue 1
Performance metrics for climate models
journal, January 2008
- Gleckler, P. J.; Taylor, K. E.; Doutriaux, C.
- Journal of Geophysical Research, Vol. 113, Issue D6
How Well Do Coupled Models Simulate Today's Climate?
journal, March 2008
- Reichler, Thomas; Kim, Junsu
- Bulletin of the American Meteorological Society, Vol. 89, Issue 3
Parametric sensitivity analysis of precipitation at global and local scales in the Community Atmosphere Model CAM5
journal, April 2015
- Qian, Yun; Yan, Huiping; Hou, Zhangshuan
- Journal of Advances in Modeling Earth Systems, Vol. 7, Issue 2
The Art and Science of Climate Model Tuning
journal, March 2017
- Hourdin, Frédéric; Mauritsen, Thorsten; Gettelman, Andrew
- Bulletin of the American Meteorological Society, Vol. 98, Issue 3
A likelihood-based comparison of temporal models for physical processes
journal, April 2011
- Braverman, Amy; Cressie, Noel; Teixeira, Joao
- Statistical Analysis and Data Mining, Vol. 4, Issue 3
An automatic and effective parameter optimization method for model tuning
journal, January 2015
- Zhang, T.; Li, L.; Lin, Y.
- Geoscientific Model Development, Vol. 8, Issue 11
A Bayesian approach to climate model evaluation and multi-model averaging with an application to global mean surface temperatures from IPCC AR4 coupled climate models
journal, January 2006
- Min, Seung-Ki; Hense, Andreas
- Geophysical Research Letters, Vol. 33, Issue 8
An Evaluation of Decadal Probability Forecasts from State-of-the-Art Climate Models
journal, December 2013
- Suckling, Emma B.; Smith, Leonard A.
- Journal of Climate, Vol. 26, Issue 23
A new test statistic for climate models that includes field and spatial dependencies using Gaussian Markov random fields
journal, January 2016
- Nosedal-Sanchez, Alvaro; Jackson, Charles S.; Huerta, Gabriel
- Geoscientific Model Development, Vol. 9, Issue 7
Uncertainty quantification and parameter tuning in the CAM5 Zhang-McFarlane convection scheme and impact of improved convection on the global circulation and climate: PARAMETERS TUNING IN CAM5 AND ITS IMPACT
journal, January 2013
- Yang, Ben; Qian, Yun; Lin, Guang
- Journal of Geophysical Research: Atmospheres, Vol. 118, Issue 2
Works referencing / citing this record:
A likelihood-based comparison of temporal models for physical processes
journal, April 2011
- Braverman, Amy; Cressie, Noel; Teixeira, Joao
- Statistical Analysis and Data Mining, Vol. 4, Issue 3
Aerosols implicated as a prime driver of twentieth-century North Atlantic climate variability
journal, April 2012
- Booth, Ben B. B.; Dunstone, Nick J.; Halloran, Paul R.
- Nature, Vol. 484, Issue 7393
Improving our fundamental understanding of the role of aerosol−cloud interactions in the climate system
journal, May 2016
- Seinfeld, John H.; Bretherton, Christopher; Carslaw, Kenneth S.
- Proceedings of the National Academy of Sciences, Vol. 113, Issue 21
Measuring Agreement about Ranked Decision Choices for a Single Subject
journal, January 2009
- Riffenburgh, Robert H.; Johnstone, Peter A.
- The International Journal of Biostatistics, Vol. 5, Issue 1
Limitations of passive remote sensing to constrain global cloud condensation nuclei
journal, January 2016
- Stier, Philip
- Atmospheric Chemistry and Physics, Vol. 16, Issue 10
Selecting a climate model subset to optimise key ensemble properties
journal, January 2018
- Herger, Nadja; Abramowitz, Gab; Knutti, Reto
- Earth System Dynamics, Vol. 9, Issue 1
An automatic and effective parameter optimization method for model tuning
journal, January 2015
- Zhang, T.; Li, L.; Lin, Y.
- Geoscientific Model Development, Vol. 8, Issue 11
A new test statistic for climate models that includes field and spatial dependencies using Gaussian Markov random fields
journal, January 2016
- Nosedal-Sanchez, Alvaro; Jackson, Charles S.; Huerta, Gabriel
- Geoscientific Model Development, Vol. 9, Issue 7