The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation
Abstract
©2018. The Authors. The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted bymore »
- Authors:
-
- Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA
- Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA, Department of Civil and Environmental Engineering University of Tennessee, Knoxville Knoxville TN USA
- Climate and Global Dynamics Division National Center for Atmospheric Research Boulder CO USA
- Department of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USA
- Climate Sciences Department Lawrence Berkeley National Laboratory Berkeley CA USA
- Department of Earth System Science University of California Irvine CA USA
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Biological and Environmental Research (BER)
- OSTI Identifier:
- 1481226
- Alternate Identifier(s):
- OSTI ID: 1481227; OSTI ID: 1489599; OSTI ID: 1563970
- Grant/Contract Number:
- AC05-00OR22725; AC02-05CH11231
- Resource Type:
- Journal Article: Published Article
- Journal Name:
- Journal of Advances in Modeling Earth Systems
- Additional Journal Information:
- Journal Name: Journal of Advances in Modeling Earth Systems Journal Volume: 10 Journal Issue: 11; Journal ID: ISSN 1942-2466
- Publisher:
- American Geophysical Union (AGU)
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 54 ENVIRONMENTAL SCIENCES; benchmarking; Earth system model; model evaluation
Citation Formats
Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, and Randerson, James T. The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation. United States: N. p., 2018.
Web. doi:10.1029/2018MS001354.
Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, & Randerson, James T. The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation. United States. https://doi.org/10.1029/2018MS001354
Collier, Nathan, Hoffman, Forrest M., Lawrence, David M., Keppel‐Aleks, Gretchen, Koven, Charles D., Riley, William J., Mu, Mingquan, and Randerson, James T. 2018.
"The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation". United States. https://doi.org/10.1029/2018MS001354.
@article{osti_1481226,
title = {The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation},
author = {Collier, Nathan and Hoffman, Forrest M. and Lawrence, David M. and Keppel‐Aleks, Gretchen and Koven, Charles D. and Riley, William J. and Mu, Mingquan and Randerson, James T.},
abstractNote = {©2018. The Authors. The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted by modeling teams and centers during model development and for model intercomparison projects, and community engagement is sought for extending evaluation metrics and adding new observational data sets to the benchmarking framework.},
doi = {10.1029/2018MS001354},
url = {https://www.osti.gov/biblio/1481226},
journal = {Journal of Advances in Modeling Earth Systems},
issn = {1942-2466},
number = 11,
volume = 10,
place = {United States},
year = {Wed Nov 07 00:00:00 EST 2018},
month = {Wed Nov 07 00:00:00 EST 2018}
}
Web of Science
Works referenced in this record:
Multiple soil nutrient competition between plants, microbes, and mineral surfaces: model development, parameterization, and example applications in several tropical forests
journal, January 2016
- Zhu, Q.; Riley, W. J.; Tang, J.
- Biogeosciences, Vol. 13, Issue 1
Positive feedback between future climate change and the carbon cycle
journal, April 2001
- Friedlingstein, Pierre; Bopp, Laurent; Ciais, Philippe
- Geophysical Research Letters, Vol. 28, Issue 8
Quantifying Carbon Cycle Feedbacks
journal, October 2009
- Gregory, J. M.; Jones, C. D.; Cadule, P.
- Journal of Climate, Vol. 22, Issue 19
Comprehensive ecosystem model-data synthesis using multiple data sets at two temperate forest free-air CO 2 enrichment experiments: Model performance at ambient CO 2 concentration : FACE MODEL-DATA SYNTHESIS
journal, May 2014
- Walker, Anthony P.; Hanson, Paul J.; De Kauwe, Martin G.
- Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 5
A comprehensive benchmarking system for evaluating global vegetation models
journal, January 2013
- Kelley, D. I.; Prentice, I. C.; Harrison, S. P.
- Biogeosciences, Vol. 10, Issue 5
Global Precipitation: A 17-Year Monthly Analysis Based on Gauge Observations, Satellite Estimates, and Numerical Model Outputs
journal, November 1997
- Xie, Pingping; Arkin, Phillip A.
- Bulletin of the American Meteorological Society, Vol. 78, Issue 11
A quantitative assessment of a terrestrial biosphere model's data needs across North American biomes: PEcAn/ED model-data uncertainty analysis
journal, March 2014
- Dietze, Michael C.; Serbin, Shawn P.; Davidson, Carl
- Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 3
Recent decline in the global land evapotranspiration trend due to limited moisture supply
journal, October 2010
- Jung, Martin; Reichstein, Markus; Ciais, Philippe
- Nature, Vol. 467, Issue 7318
An Overview of CMIP5 and the Experiment Design
journal, April 2012
- Taylor, Karl E.; Stouffer, Ronald J.; Meehl, Gerald A.
- Bulletin of the American Meteorological Society, Vol. 93, Issue 4
Global Carbon Budget 2016
journal, January 2016
- Le Quéré, Corinne; Andrew, Robbie M.; Canadell, Josep G.
- Earth System Science Data, Vol. 8, Issue 2
The Plumbing of Land Surface Models: Benchmarking Model Performance
journal, June 2015
- Best, M. J.; Abramowitz, G.; Johnson, H. R.
- Journal of Hydrometeorology, Vol. 16, Issue 3
Surface Irradiances Consistent with CERES-Derived Top-of-Atmosphere Shortwave and Longwave Irradiances
journal, May 2013
- Kato, Seiji; Loeb, Norman G.; Rose, Fred G.
- Journal of Climate, Vol. 26, Issue 9
THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research
journal, September 2007
- Meehl, Gerald A.; Covey, Curt; Delworth, Thomas
- Bulletin of the American Meteorological Society, Vol. 88, Issue 9
Global land-surface evaporation estimated from satellite-based observations
journal, January 2011
- Miralles, D. G.; Holmes, T. R. H.; De Jeu, R. A. M.
- Hydrology and Earth System Sciences, Vol. 15, Issue 2
Summarizing multiple aspects of model performance in a single diagram
journal, April 2001
- Taylor, Karl E.
- Journal of Geophysical Research: Atmospheres, Vol. 106, Issue D7
Acceleration of global warming due to carbon-cycle feedbacks in a coupled climate model
journal, November 2000
- Cox, Peter M.; Betts, Richard A.; Jones, Chris D.
- Nature, Vol. 408, Issue 6809
A More Powerful Reality Test for Climate Models
journal, May 2016
- Gleckler, Peter; Doutriaux, Charles; Durack, Paul
- Eos, Vol. 97
Towards a benchmark for land surface models: TOWARDS A BENCHMARK FOR LAND SURFACE MODELS
journal, November 2005
- Abramowitz, Gab
- Geophysical Research Letters, Vol. 32, Issue 22
Towards a public, standardized, diagnostic benchmarking system for land surface models
journal, January 2012
- Abramowitz, G.
- Geoscientific Model Development, Vol. 5, Issue 3
ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP
journal, January 2016
- Eyring, Veronika; Righi, Mattia; Lauer, Axel
- Geoscientific Model Development, Vol. 9, Issue 5
A framework for benchmarking land models
journal, January 2012
- Luo, Y. Q.; Randerson, J. T.; Abramowitz, G.
- Biogeosciences, Vol. 9, Issue 10
Benchmark map of forest carbon stocks in tropical regions across three continents
journal, May 2011
- Saatchi, S. S.; Harris, N. L.; Brown, S.
- Proceedings of the National Academy of Sciences, Vol. 108, Issue 24
Causes and implications of persistent atmospheric carbon dioxide biases in Earth System Models
journal, February 2014
- Hoffman, F. M.; Randerson, J. T.; Arora, V. K.
- Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 2
Performance metrics for climate models
journal, January 2008
- Gleckler, P. J.; Taylor, K. E.; Doutriaux, C.
- Journal of Geophysical Research, Vol. 113, Issue D6
Meta-analysis of high-latitude nitrogen-addition and warming studies implies ecological mechanisms overlooked by land models
journal, January 2014
- Bouskill, N. J.; Riley, W. J.; Tang, J. Y.
- Biogeosciences, Vol. 11, Issue 23
Multicentury changes in ocean and land contributions to the climate-carbon feedback: CARBON CYCLE FEEDBACKS TO 2300 IN CESM
journal, June 2015
- Randerson, J. T.; Lindsay, K.; Munoz, E.
- Global Biogeochemical Cycles, Vol. 29, Issue 6
The ERA-Interim reanalysis: configuration and performance of the data assimilation system
journal, April 2011
- Dee, D. P.; Uppala, S. M.; Simmons, A. J.
- Quarterly Journal of the Royal Meteorological Society, Vol. 137, Issue 656
Systematic assessment of terrestrial biogeochemistry in coupled climate-carbon models
journal, October 2009
- Randerson, James T.; Hoffman, Forrest M.; Thornton, Peter E.
- Global Change Biology, Vol. 15, Issue 10
Mapping U.S. forest biomass using nationwide forest inventory data and moderate resolution information
journal, April 2008
- Blackard, J.; Finco, M.; Helmer, E.
- Remote Sensing of Environment, Vol. 112, Issue 4
Benchmarking coupled climate-carbon models against long-term atmospheric CO 2 measurements : COUPLED CLIMATE-CARBON MODELS BENCHMARKS
journal, June 2010
- Cadule, P.; Friedlingstein, P.; Bopp, L.
- Global Biogeochemical Cycles, Vol. 24, Issue 2
Evaluation of terrestrial carbon cycle models for their response to climate variability and to CO 2 trends
journal, April 2013
- Piao, Shilong; Sitch, Stephen; Ciais, Philippe
- Global Change Biology, Vol. 19, Issue 7
Estimates of Freshwater Discharge from Continents: Latitudinal and Seasonal Variations
journal, December 2002
- Dai, Aiguo; Trenberth, Kevin E.
- Journal of Hydrometeorology, Vol. 3, Issue 6
Interactions between land use change and carbon cycle feedbacks: Land Use and Carbon Cycle Feedbacks
journal, January 2017
- Mahowald, Natalie M.; Randerson, James T.; Lindsay, Keith
- Global Biogeochemical Cycles, Vol. 31, Issue 1
Uncertainties in CMIP5 Climate Projections due to Carbon Cycle Feedbacks
journal, January 2014
- Friedlingstein, Pierre; Meinshausen, Malte; Arora, Vivek K.
- Journal of Climate, Vol. 27, Issue 2
GPCC's new land surface precipitation climatology based on quality-controlled in situ data and its role in quantifying the global water cycle
journal, March 2013
- Schneider, Udo; Becker, Andreas; Finger, Peter
- Theoretical and Applied Climatology, Vol. 115, Issue 1-2
Evaluation of 11 terrestrial carbon-nitrogen cycle models against observations from two temperate Free-Air CO 2 Enrichment studies
journal, January 2014
- Zaehle, Sönke; Medlyn, Belinda E.; De Kauwe, Martin G.
- New Phytologist, Vol. 202, Issue 3
Predicting long-term carbon sequestration in response to CO 2 enrichment: How and why do current ecosystem models differ?
journal, April 2015
- Walker, Anthony P.; Zaehle, Sönke; Medlyn, Belinda E.
- Global Biogeochemical Cycles, Vol. 29, Issue 4
Carbon–Concentration and Carbon–Climate Feedbacks in CMIP5 Earth System Models
journal, August 2013
- Arora, Vivek K.; Boer, George J.; Friedlingstein, Pierre
- Journal of Climate, Vol. 26, Issue 15
How Well Do Coupled Models Simulate Today's Climate?
journal, March 2008
- Reichler, Thomas; Kim, Junsu
- Bulletin of the American Meteorological Society, Vol. 89, Issue 3
Representing leaf and root physiological traits in CLM improves global carbon and nitrogen cycling predictions: LEAF AND ROOT TRAITS IN CLM
journal, May 2016
- Ghimire, Bardan; Riley, William J.; Koven, Charles D.
- Journal of Advances in Modeling Earth Systems, Vol. 8, Issue 2
A new data set for estimating organic carbon storage to 3 m depth in soils of the northern circumpolar permafrost region
journal, January 2013
- Hugelius, G.; Bockheim, J. G.; Camill, P.
- Earth System Science Data, Vol. 5, Issue 2
Causes of variation in soil carbon simulations from CMIP5 Earth system models and comparison with observations
journal, January 2013
- Todd-Brown, K. E. O.; Randerson, J. T.; Post, W. M.
- Biogeosciences, Vol. 10, Issue 3
Towards a more objective evaluation of modelled land-carbon trends using atmospheric CO 2 and satellite-based vegetation activity observations
journal, January 2013
- Dalmonech, D.; Zaehle, S.
- Biogeosciences, Vol. 10, Issue 6
Comparing Evaporative Sources of Terrestrial Precipitation and Their Extremes in MERRA Using Relative Entropy
journal, February 2014
- Dirmeyer, Paul A.; Wei, Jiangfeng; Bosilovich, Michael G.
- Journal of Hydrometeorology, Vol. 15, Issue 1
Climate–Carbon Cycle Feedback Analysis: Results from the C 4 MIP Model Intercomparison
journal, July 2006
- Friedlingstein, P.; Cox, P.; Betts, R.
- Journal of Climate, Vol. 19, Issue 14
Estimation of global leaf area index and absorbed par using radiative transfer models
journal, January 1997
- Myneni, R. B.; Ramakrishna, R.; Nemani, R.
- IEEE Transactions on Geoscience and Remote Sensing, Vol. 35, Issue 6
Estimating Climatological Bias Errors for the Global Precipitation Climatology Project (GPCP)
journal, January 2012
- Adler, Robert F.; Gu, Guojun; Huffman, George J.
- Journal of Applied Meteorology and Climatology, Vol. 51, Issue 1
A comprehensive set of benchmark tests for a land surface model of simultaneous fluxes of water and carbon at both the global and seasonal scale
journal, January 2011
- Blyth, E.; Clark, D. B.; Ellis, R.
- Geoscientific Model Development, Vol. 4, Issue 2
Evaluating the Land and Ocean Components of the Global Carbon Cycle in the CMIP5 Earth System Models
journal, September 2013
- Anav, A.; Friedlingstein, P.; Kidston, M.
- Journal of Climate, Vol. 26, Issue 18
Separation of net ecosystem exchange into assimilation and respiration using a light response curve approach: critical issues and global evaluation: SEPARATION OF NEE INTO GPP AND RECO
journal, January 2010
- Lasslop, Gitta; Reichstein, Markus; Papale, Dario
- Global Change Biology, Vol. 16, Issue 1
Sustained climate warming drives declining marine biological productivity
journal, March 2018
- Moore, J. Keith; Fu, Weiwei; Primeau, Francois
- Science, Vol. 359, Issue 6380
Evolution of carbon sinks in a changing climate
journal, August 2005
- Fung, I. Y.; Doney, S. C.; Lindsay, K.
- Proceedings of the National Academy of Sciences, Vol. 102, Issue 32
Assessing variability and long-term trends in burned area by merging multiple satellite fire products
journal, January 2010
- Giglio, L.; Randerson, J. T.; van der Werf, G. R.
- Biogeosciences, Vol. 7, Issue 3
Facilitating feedbacks between field measurements and ecosystem models
journal, May 2013
- LeBauer, David S.; Wang, Dan; Richter, Katherine T.
- Ecological Monographs, Vol. 83, Issue 2
Post-processing removal of correlated errors in GRACE data
journal, January 2006
- Swenson, Sean; Wahr, John
- Geophysical Research Letters, Vol. 33, Issue 8
An assessment of the MODIS collection 5 leaf area index product for a region of mixed coniferous forest
journal, February 2011
- De Kauwe, M. G.; Disney, M. I.; Quaife, T.
- Remote Sensing of Environment, Vol. 115, Issue 2
Updated high-resolution grids of monthly climatic observations - the CRU TS3.10 Dataset: UPDATED HIGH-RESOLUTION GRIDS OF MONTHLY CLIMATIC OBSERVATIONS
journal, May 2013
- Harris, I.; Jones, P. D.; Osborn, T. J.
- International Journal of Climatology, Vol. 34, Issue 3
Land surface Verification Toolkit (LVT) – a generalized framework for land surface model evaluation
journal, January 2012
- Kumar, S. V.; Peters-Lidard, C. D.; Santanello, J.
- Geoscientific Model Development, Vol. 5, Issue 3