skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

Abstract

©2018. The Authors. The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted bymore » modeling teams and centers during model development and for model intercomparison projects, and community engagement is sought for extending evaluation metrics and adding new observational data sets to the benchmarking framework.« less

Authors:
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3]; ORCiD logo [4]; ORCiD logo [5]; ORCiD logo [5];  [6]; ORCiD logo [6]
  1. Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA
  2. Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA, Department of Civil and Environmental Engineering University of Tennessee, Knoxville Knoxville TN USA
  3. Climate and Global Dynamics Division National Center for Atmospheric Research Boulder CO USA
  4. Department of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USA
  5. Climate Sciences Department Lawrence Berkeley National Laboratory Berkeley CA USA
  6. Department of Earth System Science University of California Irvine CA USA
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER)
OSTI Identifier:
1481226
Alternate Identifier(s):
OSTI ID: 1481227; OSTI ID: 1489599; OSTI ID: 1563970
Grant/Contract Number:  
AC05-00OR22725; AC02-05CH11231
Resource Type:
Journal Article: Published Article
Journal Name:
Journal of Advances in Modeling Earth Systems
Additional Journal Information:
Journal Name: Journal of Advances in Modeling Earth Systems Journal Volume: 10 Journal Issue: 11; Journal ID: ISSN 1942-2466
Publisher:
American Geophysical Union (AGU)
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; benchmarking; Earth system model; model evaluation

Citation Formats

Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, and Randerson, James T. The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation. United States: N. p., 2018. Web. doi:10.1029/2018MS001354.
Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, & Randerson, James T. The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation. United States. https://doi.org/10.1029/2018MS001354
Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, and Randerson, James T. 2018. "The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation". United States. https://doi.org/10.1029/2018MS001354.
@article{osti_1481226,
title = {The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation},
author = {Collier, Nathan and Hoffman, Forrest M. and Lawrence, David M. and Keppel‐Aleks, Gretchen and Koven, Charles D. and Riley, William J. and Mu, Mingquan and Randerson, James T.},
abstractNote = {©2018. The Authors. The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted by modeling teams and centers during model development and for model intercomparison projects, and community engagement is sought for extending evaluation metrics and adding new observational data sets to the benchmarking framework.},
doi = {10.1029/2018MS001354},
url = {https://www.osti.gov/biblio/1481226}, journal = {Journal of Advances in Modeling Earth Systems},
issn = {1942-2466},
number = 11,
volume = 10,
place = {United States},
year = {Wed Nov 07 00:00:00 EST 2018},
month = {Wed Nov 07 00:00:00 EST 2018}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record at https://doi.org/10.1029/2018MS001354

Citation Metrics:
Cited by: 128 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Positive feedback between future climate change and the carbon cycle
journal, April 2001


Quantifying Carbon Cycle Feedbacks
journal, October 2009


A comprehensive benchmarking system for evaluating global vegetation models
journal, January 2013


Global Precipitation: A 17-Year Monthly Analysis Based on Gauge Observations, Satellite Estimates, and Numerical Model Outputs
journal, November 1997


A quantitative assessment of a terrestrial biosphere model's data needs across North American biomes: PEcAn/ED model-data uncertainty analysis
journal, March 2014


Recent decline in the global land evapotranspiration trend due to limited moisture supply
journal, October 2010


An Overview of CMIP5 and the Experiment Design
journal, April 2012


Global Carbon Budget 2016
journal, January 2016


The Plumbing of Land Surface Models: Benchmarking Model Performance
journal, June 2015


Surface Irradiances Consistent with CERES-Derived Top-of-Atmosphere Shortwave and Longwave Irradiances
journal, May 2013


THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research
journal, September 2007


Global land-surface evaporation estimated from satellite-based observations
journal, January 2011


Summarizing multiple aspects of model performance in a single diagram
journal, April 2001


Acceleration of global warming due to carbon-cycle feedbacks in a coupled climate model
journal, November 2000


A More Powerful Reality Test for Climate Models
journal, May 2016


Towards a benchmark for land surface models: TOWARDS A BENCHMARK FOR LAND SURFACE MODELS
journal, November 2005


Towards a public, standardized, diagnostic benchmarking system for land surface models
journal, January 2012


ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
journal, January 2016


A framework for benchmarking land models
journal, January 2012


Benchmark map of forest carbon stocks in tropical regions across three continents
journal, May 2011


Causes and implications of persistent atmospheric carbon dioxide biases in Earth System Models
journal, February 2014


Performance metrics for climate models
journal, January 2008


Multicentury changes in ocean and land contributions to the climate-carbon feedback: CARBON CYCLE FEEDBACKS TO 2300 IN CESM
journal, June 2015


The ERA-Interim reanalysis: configuration and performance of the data assimilation system
journal, April 2011


Systematic assessment of terrestrial biogeochemistry in coupled climate-carbon models
journal, October 2009


Mapping U.S. forest biomass using nationwide forest inventory data and moderate resolution information
journal, April 2008


Evaluation of terrestrial carbon cycle models for their response to climate variability and to CO 2 trends
journal, April 2013


Estimates of Freshwater Discharge from Continents: Latitudinal and Seasonal Variations
journal, December 2002


Interactions between land use change and carbon cycle feedbacks: Land Use and Carbon Cycle Feedbacks
journal, January 2017


Uncertainties in CMIP5 Climate Projections due to Carbon Cycle Feedbacks
journal, January 2014


GPCC's new land surface precipitation climatology based on quality-controlled in situ data and its role in quantifying the global water cycle
journal, March 2013


Evaluation of 11 terrestrial carbon-nitrogen cycle models against observations from two temperate Free-Air CO 2 Enrichment studies
journal, January 2014


Predicting long-term carbon sequestration in response to CO 2 enrichment: How and why do current ecosystem models differ?
journal, April 2015


Carbon–Concentration and Carbon–Climate Feedbacks in CMIP5 Earth System Models
journal, August 2013


How Well Do Coupled Models Simulate Today's Climate?
journal, March 2008


Representing leaf and root physiological traits in CLM improves global carbon and nitrogen cycling predictions: LEAF AND ROOT TRAITS IN CLM
journal, May 2016


A new data set for estimating organic carbon storage to 3 m depth in soils of the northern circumpolar permafrost region
journal, January 2013


Causes of variation in soil carbon simulations from CMIP5 Earth system models and comparison with observations
journal, January 2013


Comparing Evaporative Sources of Terrestrial Precipitation and Their Extremes in MERRA Using Relative Entropy
journal, February 2014


Climate–Carbon Cycle Feedback Analysis: Results from the C 4 MIP Model Intercomparison
journal, July 2006


Estimation of global leaf area index and absorbed par using radiative transfer models
journal, January 1997


Estimating Climatological Bias Errors for the Global Precipitation Climatology Project (GPCP)
journal, January 2012


Evaluating the Land and Ocean Components of the Global Carbon Cycle in the CMIP5 Earth System Models
journal, September 2013


Sustained climate warming drives declining marine biological productivity
journal, March 2018


Evolution of carbon sinks in a changing climate
journal, August 2005


Assessing variability and long-term trends in burned area by merging multiple satellite fire products
journal, January 2010


Facilitating feedbacks between field measurements and ecosystem models
journal, May 2013


Post-processing removal of correlated errors in GRACE data
journal, January 2006


An assessment of the MODIS collection 5 leaf area index product for a region of mixed coniferous forest
journal, February 2011


Land surface Verification Toolkit (LVT) – a generalized framework for land surface model evaluation
journal, January 2012