The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

Collier, Nathan; Hoffman, Forrest M.; Lawrence, David M.; Keppel‐Aleks, Gretchen; Koven, Charles D.; Riley, William J.; Mu, Mingquan; Randerson, James T.

doi:10.1029/2018MS001354

Title: The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

Journal Article · Wed Nov 07 00:00:00 EST 2018 · Journal of Advances in Modeling Earth Systems

DOI:https://doi.org/10.1029/2018MS001354· OSTI ID:1481226

^[1];

^[2];

^[3];

^[4];

^[5];

^[5]; Mu, Mingquan ^[6];

^[6]

Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA
Climate Change Science Institute Oak Ridge National Laboratory Oak Ridge TN USA, Department of Civil and Environmental Engineering University of Tennessee, Knoxville Knoxville TN USA
Climate and Global Dynamics Division National Center for Atmospheric Research Boulder CO USA
Department of Climate and Space Sciences and Engineering University of Michigan Ann Arbor MI USA
Climate Sciences Department Lawrence Berkeley National Laboratory Berkeley CA USA
Department of Earth System Science University of California Irvine CA USA

©2018. The Authors. The increasing complexity of Earth system models has inspired efforts to quantitatively assess model fidelity through rigorous comparison with best available measurements and observational data products. Earth system models exhibit a high degree of spread in predictions of land biogeochemistry, biogeophysics, and hydrology, which are sensitive to forcing from other model components. Based on insights from prior land model evaluation studies and community workshops, the authors developed an open source model benchmarking software package that generates graphical diagnostics and scores model performance in support of the International Land Model Benchmarking (ILAMB) project. Employing a suite of in situ, remote sensing, and reanalysis data sets, the ILAMB package performs comprehensive model assessment across a wide range of land variables and generates a hierarchical set of web pages containing statistical analyses and figures designed to provide the user insights into strengths and weaknesses of multiple models or model versions. Described here is the benchmarking philosophy and mathematical methodology embodied in the most recent implementation of the ILAMB package. Comparison methods unique to a few specific data sets are presented, and guidelines for configuring an ILAMB analysis and interpreting resulting model performance scores are discussed. ILAMB is being adopted by modeling teams and centers during model development and for model intercomparison projects, and community engagement is sought for extending evaluation metrics and adding new observational data sets to the benchmarking framework.

View Journal Article

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER)

Grant/Contract Number:: AC05-00OR22725; AC02-05CH11231

OSTI ID:: 1481226

Alternate ID(s):: OSTI ID: 1481227; OSTI ID: 1489599; OSTI ID: 1563970

Journal Information:: Journal of Advances in Modeling Earth Systems, Journal Name: Journal of Advances in Modeling Earth Systems Vol. 10 Journal Issue: 11; ISSN 1942-2466

Publisher:: American Geophysical Union (AGU)Copyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 144 works

Citation information provided by
Web of Science

References (59)

Multiple soil nutrient competition between plants, microbes, and mineral surfaces: model development, parameterization, and example applications in several tropical forests Zhu, Q.; Riley, W. J.; Tang, J. Biogeosciences, Vol. 13, Issue 1 https://doi.org/10.5194/bg-13-341-2016	journal	January 2016
Positive feedback between future climate change and the carbon cycle Friedlingstein, Pierre; Bopp, Laurent; Ciais, Philippe Geophysical Research Letters, Vol. 28, Issue 8 https://doi.org/10.1029/2000GL012015	journal	April 2001
Quantifying Carbon Cycle Feedbacks Gregory, J. M.; Jones, C. D.; Cadule, P. Journal of Climate, Vol. 22, Issue 19 https://doi.org/10.1175/2009JCLI2949.1	journal	October 2009
Comprehensive ecosystem model-data synthesis using multiple data sets at two temperate forest free-air CO ₂ enrichment experiments: Model performance at ambient CO ₂ concentration : FACE MODEL-DATA SYNTHESIS Walker, Anthony P.; Hanson, Paul J.; De Kauwe, Martin G. Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 5 https://doi.org/10.1002/2013JG002553	journal	May 2014
A comprehensive benchmarking system for evaluating global vegetation models Kelley, D. I.; Prentice, I. C.; Harrison, S. P. Biogeosciences, Vol. 10, Issue 5 https://doi.org/10.5194/bg-10-3313-2013	journal	January 2013
Global Precipitation: A 17-Year Monthly Analysis Based on Gauge Observations, Satellite Estimates, and Numerical Model Outputs Xie, Pingping; Arkin, Phillip A. Bulletin of the American Meteorological Society, Vol. 78, Issue 11 https://doi.org/10.1175/1520-0477(1997)078<2539:GPAYMA>2.0.CO;2	journal	November 1997
A quantitative assessment of a terrestrial biosphere model's data needs across North American biomes: PEcAn/ED model-data uncertainty analysis Dietze, Michael C.; Serbin, Shawn P.; Davidson, Carl Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 3 https://doi.org/10.1002/2013JG002392	journal	March 2014
Recent decline in the global land evapotranspiration trend due to limited moisture supply Jung, Martin; Reichstein, Markus; Ciais, Philippe Nature, Vol. 467, Issue 7318 https://doi.org/10.1038/nature09396	journal	October 2010
An Overview of CMIP5 and the Experiment Design Taylor, Karl E.; Stouffer, Ronald J.; Meehl, Gerald A. Bulletin of the American Meteorological Society, Vol. 93, Issue 4 https://doi.org/10.1175/BAMS-D-11-00094.1	journal	April 2012
Global Carbon Budget 2016 Le Quéré, Corinne; Andrew, Robbie M.; Canadell, Josep G. Earth System Science Data, Vol. 8, Issue 2 https://doi.org/10.5194/essd-8-605-2016	journal	January 2016
The Plumbing of Land Surface Models: Benchmarking Model Performance Best, M. J.; Abramowitz, G.; Johnson, H. R. Journal of Hydrometeorology, Vol. 16, Issue 3 https://doi.org/10.1175/JHM-D-14-0158.1	journal	June 2015
Surface Irradiances Consistent with CERES-Derived Top-of-Atmosphere Shortwave and Longwave Irradiances Kato, Seiji; Loeb, Norman G.; Rose, Fred G. Journal of Climate, Vol. 26, Issue 9 https://doi.org/10.1175/JCLI-D-12-00436.1	journal	May 2013
THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research Meehl, Gerald A.; Covey, Curt; Delworth, Thomas Bulletin of the American Meteorological Society, Vol. 88, Issue 9 https://doi.org/10.1175/BAMS-88-9-1383	journal	September 2007
Global land-surface evaporation estimated from satellite-based observations Miralles, D. G.; Holmes, T. R. H.; De Jeu, R. A. M. Hydrology and Earth System Sciences, Vol. 15, Issue 2 https://doi.org/10.5194/hess-15-453-2011	journal	January 2011
Summarizing multiple aspects of model performance in a single diagram Taylor, Karl E. Journal of Geophysical Research: Atmospheres, Vol. 106, Issue D7 https://doi.org/10.1029/2000JD900719	journal	April 2001
Acceleration of global warming due to carbon-cycle feedbacks in a coupled climate model Cox, Peter M.; Betts, Richard A.; Jones, Chris D. Nature, Vol. 408, Issue 6809 https://doi.org/10.1038/35041539	journal	November 2000
A More Powerful Reality Test for Climate Models Gleckler, Peter; Doutriaux, Charles; Durack, Paul Eos, Vol. 97 https://doi.org/10.1029/2016EO051663	journal	May 2016
Towards a benchmark for land surface models: TOWARDS A BENCHMARK FOR LAND SURFACE MODELS Abramowitz, Gab Geophysical Research Letters, Vol. 32, Issue 22 https://doi.org/10.1029/2005GL024419	journal	November 2005
Towards a public, standardized, diagnostic benchmarking system for land surface models Abramowitz, G. Geoscientific Model Development, Vol. 5, Issue 3 https://doi.org/10.5194/gmd-5-819-2012	journal	January 2012
ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP Eyring, Veronika; Righi, Mattia; Lauer, Axel Geoscientific Model Development, Vol. 9, Issue 5 https://doi.org/10.5194/gmd-9-1747-2016	journal	January 2016
A framework for benchmarking land models Luo, Y. Q.; Randerson, J. T.; Abramowitz, G. Biogeosciences, Vol. 9, Issue 10 https://doi.org/10.5194/bg-9-3857-2012	journal	January 2012
Benchmark map of forest carbon stocks in tropical regions across three continents Saatchi, S. S.; Harris, N. L.; Brown, S. Proceedings of the National Academy of Sciences, Vol. 108, Issue 24 https://doi.org/10.1073/pnas.1019576108	journal	May 2011
Causes and implications of persistent atmospheric carbon dioxide biases in Earth System Models Hoffman, F. M.; Randerson, J. T.; Arora, V. K. Journal of Geophysical Research: Biogeosciences, Vol. 119, Issue 2 https://doi.org/10.1002/2013JG002381	journal	February 2014
Performance metrics for climate models Gleckler, P. J.; Taylor, K. E.; Doutriaux, C. Journal of Geophysical Research, Vol. 113, Issue D6 https://doi.org/10.1029/2007JD008972	journal	January 2008
Meta-analysis of high-latitude nitrogen-addition and warming studies implies ecological mechanisms overlooked by land models Bouskill, N. J.; Riley, W. J.; Tang, J. Y. Biogeosciences, Vol. 11, Issue 23 https://doi.org/10.5194/bg-11-6969-2014	journal	January 2014
Multicentury changes in ocean and land contributions to the climate-carbon feedback: CARBON CYCLE FEEDBACKS TO 2300 IN CESM Randerson, J. T.; Lindsay, K.; Munoz, E. Global Biogeochemical Cycles, Vol. 29, Issue 6 https://doi.org/10.1002/2014GB005079	journal	June 2015
The ERA-Interim reanalysis: configuration and performance of the data assimilation system Dee, D. P.; Uppala, S. M.; Simmons, A. J. Quarterly Journal of the Royal Meteorological Society, Vol. 137, Issue 656 https://doi.org/10.1002/qj.828	journal	April 2011
Systematic assessment of terrestrial biogeochemistry in coupled climate-carbon models Randerson, James T.; Hoffman, Forrest M.; Thornton, Peter E. Global Change Biology, Vol. 15, Issue 10 https://doi.org/10.1111/j.1365-2486.2009.01912.x	journal	October 2009
Mapping U.S. forest biomass using nationwide forest inventory data and moderate resolution information Blackard, J.; Finco, M.; Helmer, E. Remote Sensing of Environment, Vol. 112, Issue 4 https://doi.org/10.1016/j.rse.2007.08.021	journal	April 2008
Benchmarking coupled climate-carbon models against long-term atmospheric CO ₂ measurements : COUPLED CLIMATE-CARBON MODELS BENCHMARKS Cadule, P.; Friedlingstein, P.; Bopp, L. Global Biogeochemical Cycles, Vol. 24, Issue 2 https://doi.org/10.1029/2009GB003556	journal	June 2010
Evaluation of terrestrial carbon cycle models for their response to climate variability and to CO ₂ trends Piao, Shilong; Sitch, Stephen; Ciais, Philippe Global Change Biology, Vol. 19, Issue 7 https://doi.org/10.1111/gcb.12187	journal	April 2013
Estimates of Freshwater Discharge from Continents: Latitudinal and Seasonal Variations Dai, Aiguo; Trenberth, Kevin E. Journal of Hydrometeorology, Vol. 3, Issue 6 https://doi.org/10.1175/1525-7541(2002)003<0660:EOFDFC>2.0.CO;2	journal	December 2002
Interactions between land use change and carbon cycle feedbacks: Land Use and Carbon Cycle Feedbacks Mahowald, Natalie M.; Randerson, James T.; Lindsay, Keith Global Biogeochemical Cycles, Vol. 31, Issue 1 https://doi.org/10.1002/2016GB005374	journal	January 2017
Uncertainties in CMIP5 Climate Projections due to Carbon Cycle Feedbacks Friedlingstein, Pierre; Meinshausen, Malte; Arora, Vivek K. Journal of Climate, Vol. 27, Issue 2 https://doi.org/10.1175/JCLI-D-12-00579.1	journal	January 2014
GPCC's new land surface precipitation climatology based on quality-controlled in situ data and its role in quantifying the global water cycle Schneider, Udo; Becker, Andreas; Finger, Peter Theoretical and Applied Climatology, Vol. 115, Issue 1-2 https://doi.org/10.1007/s00704-013-0860-x	journal	March 2013
Evaluation of 11 terrestrial carbon-nitrogen cycle models against observations from two temperate Free-Air CO ₂ Enrichment studies Zaehle, Sönke; Medlyn, Belinda E.; De Kauwe, Martin G. New Phytologist, Vol. 202, Issue 3 https://doi.org/10.1111/nph.12697	journal	January 2014
Predicting long-term carbon sequestration in response to CO ₂ enrichment: How and why do current ecosystem models differ? Walker, Anthony P.; Zaehle, Sönke; Medlyn, Belinda E. Global Biogeochemical Cycles, Vol. 29, Issue 4 https://doi.org/10.1002/2014GB004995	journal	April 2015
Carbon–Concentration and Carbon–Climate Feedbacks in CMIP5 Earth System Models Arora, Vivek K.; Boer, George J.; Friedlingstein, Pierre Journal of Climate, Vol. 26, Issue 15 https://doi.org/10.1175/JCLI-D-12-00494.1	journal	August 2013
How Well Do Coupled Models Simulate Today's Climate? Reichler, Thomas; Kim, Junsu Bulletin of the American Meteorological Society, Vol. 89, Issue 3 https://doi.org/10.1175/BAMS-89-3-303	journal	March 2008
Representing leaf and root physiological traits in CLM improves global carbon and nitrogen cycling predictions: LEAF AND ROOT TRAITS IN CLM Ghimire, Bardan; Riley, William J.; Koven, Charles D. Journal of Advances in Modeling Earth Systems, Vol. 8, Issue 2 https://doi.org/10.1002/2015MS000538	journal	May 2016
A new data set for estimating organic carbon storage to 3 m depth in soils of the northern circumpolar permafrost region Hugelius, G.; Bockheim, J. G.; Camill, P. Earth System Science Data, Vol. 5, Issue 2 https://doi.org/10.5194/essd-5-393-2013	journal	January 2013
Causes of variation in soil carbon simulations from CMIP5 Earth system models and comparison with observations Todd-Brown, K. E. O.; Randerson, J. T.; Post, W. M. Biogeosciences, Vol. 10, Issue 3 https://doi.org/10.5194/bg-10-1717-2013	journal	January 2013
Towards a more objective evaluation of modelled land-carbon trends using atmospheric CO ₂ and satellite-based vegetation activity observations Dalmonech, D.; Zaehle, S. Biogeosciences, Vol. 10, Issue 6 https://doi.org/10.5194/bg-10-4189-2013	journal	January 2013
Comparing Evaporative Sources of Terrestrial Precipitation and Their Extremes in MERRA Using Relative Entropy Dirmeyer, Paul A.; Wei, Jiangfeng; Bosilovich, Michael G. Journal of Hydrometeorology, Vol. 15, Issue 1 https://doi.org/10.1175/JHM-D-13-053.1	journal	February 2014
Climate–Carbon Cycle Feedback Analysis: Results from the C ⁴ MIP Model Intercomparison Friedlingstein, P.; Cox, P.; Betts, R. Journal of Climate, Vol. 19, Issue 14 https://doi.org/10.1175/JCLI3800.1	journal	July 2006
Estimation of global leaf area index and absorbed par using radiative transfer models Myneni, R. B.; Ramakrishna, R.; Nemani, R. IEEE Transactions on Geoscience and Remote Sensing, Vol. 35, Issue 6 https://doi.org/10.1109/36.649788	journal	January 1997
Estimating Climatological Bias Errors for the Global Precipitation Climatology Project (GPCP) Adler, Robert F.; Gu, Guojun; Huffman, George J. Journal of Applied Meteorology and Climatology, Vol. 51, Issue 1 https://doi.org/10.1175/JAMC-D-11-052.1	journal	January 2012
A comprehensive set of benchmark tests for a land surface model of simultaneous fluxes of water and carbon at both the global and seasonal scale Blyth, E.; Clark, D. B.; Ellis, R. Geoscientific Model Development, Vol. 4, Issue 2 https://doi.org/10.5194/gmd-4-255-2011	journal	January 2011
Evaluating the Land and Ocean Components of the Global Carbon Cycle in the CMIP5 Earth System Models Anav, A.; Friedlingstein, P.; Kidston, M. Journal of Climate, Vol. 26, Issue 18 https://doi.org/10.1175/JCLI-D-12-00417.1	journal	September 2013
Separation of net ecosystem exchange into assimilation and respiration using a light response curve approach: critical issues and global evaluation: SEPARATION OF NEE INTO GPP AND RECO Lasslop, Gitta; Reichstein, Markus; Papale, Dario Global Change Biology, Vol. 16, Issue 1 https://doi.org/10.1111/j.1365-2486.2009.02041.x	journal	January 2010
Sustained climate warming drives declining marine biological productivity Moore, J. Keith; Fu, Weiwei; Primeau, Francois Science, Vol. 359, Issue 6380 https://doi.org/10.1126/science.aao6379	journal	March 2018
Evolution of carbon sinks in a changing climate Fung, I. Y.; Doney, S. C.; Lindsay, K. Proceedings of the National Academy of Sciences, Vol. 102, Issue 32 https://doi.org/10.1073/pnas.0504949102	journal	August 2005
Assessing variability and long-term trends in burned area by merging multiple satellite fire products Giglio, L.; Randerson, J. T.; van der Werf, G. R. Biogeosciences, Vol. 7, Issue 3 https://doi.org/10.5194/bg-7-1171-2010	journal	January 2010
Facilitating feedbacks between field measurements and ecosystem models LeBauer, David S.; Wang, Dan; Richter, Katherine T. Ecological Monographs, Vol. 83, Issue 2 https://doi.org/10.1890/12-0137.1	journal	May 2013
Post-processing removal of correlated errors in GRACE data Swenson, Sean; Wahr, John Geophysical Research Letters, Vol. 33, Issue 8 https://doi.org/10.1029/2005GL025285	journal	January 2006
An assessment of the MODIS collection 5 leaf area index product for a region of mixed coniferous forest De Kauwe, M. G.; Disney, M. I.; Quaife, T. Remote Sensing of Environment, Vol. 115, Issue 2 https://doi.org/10.1016/j.rse.2010.11.004	journal	February 2011
Updated high-resolution grids of monthly climatic observations - the CRU TS3.10 Dataset: UPDATED HIGH-RESOLUTION GRIDS OF MONTHLY CLIMATIC OBSERVATIONS Harris, I.; Jones, P. D.; Osborn, T. J. International Journal of Climatology, Vol. 34, Issue 3 https://doi.org/10.1002/joc.3711	journal	May 2013
Data Assimilation: A Mathematical Introduction Law, Kody; Stuart, Andrew; Zygalakis, Konstantinos Texts in Applied Mathematics https://doi.org/10.1007/978-3-319-20325-6	book	January 2015
Land surface Verification Toolkit (LVT) – a generalized framework for land surface model evaluation Kumar, S. V.; Peters-Lidard, C. D.; Santanello, J. Geoscientific Model Development, Vol. 5, Issue 3 https://doi.org/10.5194/gmd-5-869-2012	journal	January 2012

Similar Records

An international land-biosphere model benchmarking activity for the IPCC Fifth Assessment Report (AR5)

Conference · Tue Dec 01 00:00:00 EST 2009 · OSTI ID:1481226

Hoffman, Forrest M; Randerson, James T; Thornton, Peter E; +3 more

International land Model Benchmarking (ILAMB) Package v001.00

Dataset · Mon May 02 00:00:00 EDT 2016 · OSTI ID:1481226

Mu, Mingquan; Randerson, James T.; Riley, William J.; +1 more

International land Model Benchmarking (ILAMB) Package v002.00

Dataset · Mon May 09 00:00:00 EDT 2016 · OSTI ID:1481226

Collier, Nathaniel; Hoffman, Forrest M.; Mu, Mingquan; +2 more

Related Subjects

54 ENVIRONMENTAL SCIENCES
benchmarking
Earth system model
model evaluation

Title: The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation

Citation Formats

References (59)

Similar Records

Related Subjects