skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Validation of sea ice models using an uncertainty-based distance metric for multiple model variables: NEW METRIC FOR SEA ICE MODEL VALIDATION

Abstract

Here, we implement a variance-based distance metric (D n) to objectively assess skill of sea ice models when multiple output variables or uncertainties in both model predictions and observations need to be considered. The metric compares observations and model data pairs on common spatial and temporal grids improving upon highly aggregated metrics (e.g., total sea ice extent or volume) by capturing the spatial character of model skill. The D n metric is a gamma-distributed statistic that is more general than the χ 2 statistic commonly used to assess model fit, which requires the assumption that the model is unbiased and can only incorporate observational error in the analysis. The D n statistic does not assume that the model is unbiased, and allows the incorporation of multiple observational data sets for the same variable and simultaneously for different variables, along with different types of variances that can characterize uncertainties in both observations and the model. This approach represents a step to establish a systematic framework for probabilistic validation of sea ice models. The methodology is also useful for model tuning by using the D n metric as a cost function and incorporating model parametric uncertainty as part of a scheme tomore » optimize model functionality. We apply this approach to evaluate different configurations of the standalone Los Alamos sea ice model (CICE) encompassing the parametric uncertainty in the model, and to find new sets of model configurations that produce better agreement than previous configurations between model and observational estimates of sea ice concentration and thickness.« less

Authors:
ORCiD logo [1]; ORCiD logo [1];  [2];  [2]; ORCiD logo [1];  [3];  [4]
  1. T-3 Fluid Dynamics and Solid Mechanics, Theoretical Division, Los Alamos National Laboratory, Los Alamos New Mexico USA
  2. CCS-2 Computational Physics and Methods, Computer, Computational, and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos New Mexico USA
  3. XCP-8 Verification and Analysis, X Computational Physics Division, Los Alamos National Laboratory, Los Alamos New Mexico USA
  4. Booker Scientific, Fredericksburg Texas USA
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)
OSTI Identifier:
1351197
Report Number(s):
LA-UR-16-29002
Journal ID: ISSN 2169-9275
Grant/Contract Number:
AC52-06NA25396
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Geophysical Research. Oceans
Additional Journal Information:
Journal Volume: 122; Journal Issue: 4; Journal ID: ISSN 2169-9275
Publisher:
American Geophysical Union
Country of Publication:
United States
Language:
English
Subject:
58 GEOSCIENCES; Earth Sciences; Mathematics; sea ice model, validation metrics, model uncertainty

Citation Formats

Urrego-Blanco, Jorge R., Hunke, Elizabeth C., Urban, Nathan M., Jeffery, Nicole, Turner, Adrian K., Langenbrunner, James R., and Booker, Jane M. Validation of sea ice models using an uncertainty-based distance metric for multiple model variables: NEW METRIC FOR SEA ICE MODEL VALIDATION. United States: N. p., 2017. Web. doi:10.1002/2016JC012602.
Urrego-Blanco, Jorge R., Hunke, Elizabeth C., Urban, Nathan M., Jeffery, Nicole, Turner, Adrian K., Langenbrunner, James R., & Booker, Jane M. Validation of sea ice models using an uncertainty-based distance metric for multiple model variables: NEW METRIC FOR SEA ICE MODEL VALIDATION. United States. doi:10.1002/2016JC012602.
Urrego-Blanco, Jorge R., Hunke, Elizabeth C., Urban, Nathan M., Jeffery, Nicole, Turner, Adrian K., Langenbrunner, James R., and Booker, Jane M. Sat . "Validation of sea ice models using an uncertainty-based distance metric for multiple model variables: NEW METRIC FOR SEA ICE MODEL VALIDATION". United States. doi:10.1002/2016JC012602. https://www.osti.gov/servlets/purl/1351197.
@article{osti_1351197,
title = {Validation of sea ice models using an uncertainty-based distance metric for multiple model variables: NEW METRIC FOR SEA ICE MODEL VALIDATION},
author = {Urrego-Blanco, Jorge R. and Hunke, Elizabeth C. and Urban, Nathan M. and Jeffery, Nicole and Turner, Adrian K. and Langenbrunner, James R. and Booker, Jane M.},
abstractNote = {Here, we implement a variance-based distance metric (Dn) to objectively assess skill of sea ice models when multiple output variables or uncertainties in both model predictions and observations need to be considered. The metric compares observations and model data pairs on common spatial and temporal grids improving upon highly aggregated metrics (e.g., total sea ice extent or volume) by capturing the spatial character of model skill. The Dn metric is a gamma-distributed statistic that is more general than the χ2 statistic commonly used to assess model fit, which requires the assumption that the model is unbiased and can only incorporate observational error in the analysis. The Dn statistic does not assume that the model is unbiased, and allows the incorporation of multiple observational data sets for the same variable and simultaneously for different variables, along with different types of variances that can characterize uncertainties in both observations and the model. This approach represents a step to establish a systematic framework for probabilistic validation of sea ice models. The methodology is also useful for model tuning by using the Dn metric as a cost function and incorporating model parametric uncertainty as part of a scheme to optimize model functionality. We apply this approach to evaluate different configurations of the standalone Los Alamos sea ice model (CICE) encompassing the parametric uncertainty in the model, and to find new sets of model configurations that produce better agreement than previous configurations between model and observational estimates of sea ice concentration and thickness.},
doi = {10.1002/2016JC012602},
journal = {Journal of Geophysical Research. Oceans},
number = 4,
volume = 122,
place = {United States},
year = {Sat Apr 01 00:00:00 EDT 2017},
month = {Sat Apr 01 00:00:00 EDT 2017}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:
  • To address the pressing need to better understand the behavior and complex interaction of ice sheets within the global Earth system, significant development of continental-scale, dynamical ice sheet models is underway. Concurrent to the development of the Community Ice Sheet Model (CISM), the corresponding verification and validation (V&V) process is being coordinated through a new, robust, Python-based extensible software package, the Land Ice Verification and Validation toolkit (LIVVkit). Incorporated into the typical ice sheet model development cycle, it provides robust and automated numerical verification, software verification, performance validation, and physical validation analyses on a variety of platforms, from personal laptopsmore » to the largest supercomputers. LIVVkit operates on sets of regression test and reference data sets, and provides comparisons for a suite of community prioritized tests, including configuration and parameter variations, bit-for-bit evaluation, and plots of model variables to indicate where differences occur. LIVVkit also provides an easily extensible framework to incorporate and analyze results of new intercomparison projects, new observation data, and new computing platforms. LIVVkit is designed for quick adaptation to additional ice sheet models via abstraction of model specific code, functions, and configurations into an ice sheet model description bundle outside the main LIVVkit structure. Furthermore, through shareable and accessible analysis output, LIVVkit is intended to help developers build confidence in their models and enhance the credibility of ice sheet models overall.« less
  • To address the pressing need to better understand the behavior and complex interaction of ice sheets within the global Earth system, significant development of continental-scale, dynamical ice sheet models is underway. Concurrent to the development of the Community Ice Sheet Model (CISM), the corresponding verification and validation (V&V) process is being coordinated through a new, robust, Python-based extensible software package, the Land Ice Verification and Validation toolkit (LIVVkit). Incorporated into the typical ice sheet model development cycle, it provides robust and automated numerical verification, software verification, performance validation, and physical validation analyses on a variety of platforms, from personal laptopsmore » to the largest supercomputers. LIVVkit operates on sets of regression test and reference data sets, and provides comparisons for a suite of community prioritized tests, including configuration and parameter variations, bit-for-bit evaluation, and plots of model variables to indicate where differences occur. LIVVkit also provides an easily extensible framework to incorporate and analyze results of new intercomparison projects, new observation data, and new computing platforms. LIVVkit is designed for quick adaptation to additional ice sheet models via abstraction of model specific code, functions, and configurations into an ice sheet model description bundle outside the main LIVVkit structure. Furthermore, through shareable and accessible analysis output, LIVVkit is intended to help developers build confidence in their models and enhance the credibility of ice sheet models overall.« less
  • Cited by 7