skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module

Abstract

Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable–region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may performmore » poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.« less

Authors:
 [1]; ORCiD logo [1];  [1]
  1. Pacific Northwest National Lab. (PNNL), College Park, MD (United States). Joint Global Change Research Institute
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)
OSTI Identifier:
1460015
Alternate Identifier(s):
OSTI ID: 1414538
Report Number(s):
PNNL-SA-125087
Journal ID: ISSN 1991-9603; KP1703030
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Published Article
Journal Name:
Geoscientific Model Development (Online)
Additional Journal Information:
Journal Name: Geoscientific Model Development (Online); Journal Volume: 10; Journal Issue: 12; Journal ID: ISSN 1991-9603
Publisher:
European Geosciences Union
Country of Publication:
United States
Language:
English
Subject:
58 GEOSCIENCES

Citation Formats

Snyder, Abigail C., Link, Robert P., and Calvin, Katherine V. Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module. United States: N. p., 2017. Web. doi:10.5194/gmd-10-4307-2017.
Snyder, Abigail C., Link, Robert P., & Calvin, Katherine V. Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module. United States. doi:10.5194/gmd-10-4307-2017.
Snyder, Abigail C., Link, Robert P., and Calvin, Katherine V. Wed . "Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module". United States. doi:10.5194/gmd-10-4307-2017.
@article{osti_1460015,
title = {Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module},
author = {Snyder, Abigail C. and Link, Robert P. and Calvin, Katherine V.},
abstractNote = {Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable–region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.},
doi = {10.5194/gmd-10-4307-2017},
journal = {Geoscientific Model Development (Online)},
number = 12,
volume = 10,
place = {United States},
year = {2017},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
DOI: 10.5194/gmd-10-4307-2017

Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Evaluating the use of “goodness-of-fit” Measures in hydrologic and hydroclimatic model validation
journal, January 1999

  • Legates, David R.; McCabe, Gregory J.
  • Water Resources Research, Vol. 35, Issue 1
  • DOI: 10.1029/1998WR900018

Uncertainty from Model Calibration: Applying a New Method to Transport Energy Demand Modelling
journal, August 2009

  • van Ruijven, Bas; van der Sluijs, Jeroen P.; van Vuuren, Detlef P.
  • Environmental Modeling & Assessment, Vol. 15, Issue 3
  • DOI: 10.1007/s10666-009-9200-z

Diagnostic indicators for integrated assessment models of climate policy
journal, January 2015


A framework for benchmarking land models
journal, January 2012


A model-data intercomparison of CO 2 exchange across North America: Results from the North American Carbon Program site synthesis
journal, January 2010

  • Schwalm, Christopher R.; Williams, Christopher A.; Schaefer, Kevin
  • Journal of Geophysical Research, Vol. 115
  • DOI: 10.1029/2009JG001229

Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion
journal, January 1998

  • Hurvich, Clifford M.; Simonoff, Jeffrey S.; Tsai, Chih-Ling
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 60, Issue 2
  • DOI: 10.1111/1467-9868.00125

Economic and Physical Modeling of land use in gcam 3.0 and an Application to Agricultural Productivity, Land, and Terrestrial Carbon
journal, May 2014


Validating energy-oriented CGE models
journal, September 2011


Looking back to move forward on model validation: insights from a global model of agricultural land use
journal, August 2013


How Well Do Coupled Models Simulate Today's Climate?
journal, March 2008

  • Reichler, Thomas; Kim, Junsu
  • Bulletin of the American Meteorological Society, Vol. 89, Issue 3
  • DOI: 10.1175/BAMS-89-3-303

A refined index of model performance
journal, September 2011

  • Willmott, Cort J.; Robeson, Scott M.; Matsuura, Kenji
  • International Journal of Climatology, Vol. 32, Issue 13
  • DOI: 10.1002/joc.2419

A global model for residential energy use: Uncertainty in calibration to regional data
journal, January 2010


A Hindcast Experiment Using the gcam 3.0 Agriculture and Land-Use Module
journal, February 2017


Global energy model hindcasting
journal, November 2016


A criterion of efficiency for rainfall-runoff models
journal, February 1978


River flow forecasting through conceptual models part I — A discussion of principles
journal, April 1970


The interdependence and applicability of some statistical quality measures for hydrological models
journal, April 1998


Integrated Assessment Models of Global Climate Change
journal, November 1997


Summarizing multiple aspects of model performance in a single diagram
journal, April 2001

  • Taylor, Karl E.
  • Journal of Geophysical Research: Atmospheres, Vol. 106, Issue D7
  • DOI: 10.1029/2000JD900719