skip to main content

DOE PAGESDOE PAGES

Title: Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module

Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable–region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may performmore » poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.« less
Authors:
 [1] ; ORCiD logo [1] ;  [1]
  1. Pacific Northwest National Lab. (PNNL), College Park, MD (United States). Joint Global Change Research Institute
Publication Date:
Report Number(s):
PNNL-SA-125087
Journal ID: ISSN 1991-9603; KP1703030
Grant/Contract Number:
AC05-76RL01830
Type:
Published Article
Journal Name:
Geoscientific Model Development (Online)
Additional Journal Information:
Journal Name: Geoscientific Model Development (Online); Journal Volume: 10; Journal Issue: 12; Journal ID: ISSN 1991-9603
Publisher:
European Geosciences Union
Research Org:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)
Country of Publication:
United States
Language:
English
Subject:
58 GEOSCIENCES
OSTI Identifier:
1460015
Alternate Identifier(s):
OSTI ID: 1414538

Snyder, Abigail C., Link, Robert P., and Calvin, Katherine V.. Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module. United States: N. p., Web. doi:10.5194/gmd-10-4307-2017.
Snyder, Abigail C., Link, Robert P., & Calvin, Katherine V.. Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module. United States. doi:10.5194/gmd-10-4307-2017.
Snyder, Abigail C., Link, Robert P., and Calvin, Katherine V.. 2017. "Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module". United States. doi:10.5194/gmd-10-4307-2017.
@article{osti_1460015,
title = {Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module},
author = {Snyder, Abigail C. and Link, Robert P. and Calvin, Katherine V.},
abstractNote = {Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable–region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.},
doi = {10.5194/gmd-10-4307-2017},
journal = {Geoscientific Model Development (Online)},
number = 12,
volume = 10,
place = {United States},
year = {2017},
month = {11}
}