skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Practical Use of Computationally Frugal Model Analysis Methods

Abstract

Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts

Authors:
 [1];  [2];  [3];  [4];  [5];  [6];  [7];  [8]
  1. U.S. Geological Survey, Boulder, CO (United States); Univ. of Kansas, Lawrence, KS (United States)
  2. Univ. of Adelaide, SA (Australia)
  3. National Center for Atmospheric Research, Boulder, CO (United States)
  4. Florida State Univ., Tallahassee, FL (United States)
  5. Colorado State Univ., Fort Collins, CO (United States)
  6. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  7. Univ. of Darmstadt (Germany)
  8. California State Univ. (CalState), Chico, CA (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1286771
Alternate Identifier(s):
OSTI ID: 1400448
Grant/Contract Number:  
AC05-00OR22725; SC0008272; 0911074; 21-66885
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Ground Water
Additional Journal Information:
Journal Volume: 54; Journal Issue: 2; Journal ID: ISSN 0017-467X
Publisher:
Wiley - NGWA
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; 54 ENVIRONMENTAL SCIENCES; Model calibration; Sensitivity analysis; Uncertainty; Bayesian; regression; transparency; falsifiability

Citation Formats

Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States: N. p., 2015. Web. doi:10.1111/gwat.12330.
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, & Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States. https://doi.org/10.1111/gwat.12330
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. 2015. "Practical Use of Computationally Frugal Model Analysis Methods". United States. https://doi.org/10.1111/gwat.12330. https://www.osti.gov/servlets/purl/1286771.
@article{osti_1286771,
title = {Practical Use of Computationally Frugal Model Analysis Methods},
author = {Hill, Mary C. and Kavetski, Dmitri and Clark, Martyn and Ye, Ming and Arabi, Mazdak and Lu, Dan and Foglia, Laura and Mehl, Steffen},
abstractNote = {Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts},
doi = {10.1111/gwat.12330},
url = {https://www.osti.gov/biblio/1286771}, journal = {Ground Water},
issn = {0017-467X},
number = 2,
volume = 54,
place = {United States},
year = {2015},
month = {3}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 13 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Exploring Regression Structure Using Nonparametric Functional Estimation
journal, September 1993


Uncertainty in science and its role in climate policy
journal, December 2011

  • Smith, Leonard A.; Stern, Nicholas
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 369, Issue 1956
  • https://doi.org/10.1098/rsta.2011.0149

Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water: OPINION
journal, May 2011


Testing Alternative Ground Water Models Using Cross-Validation and Other Methods
journal, September 2007


Two statistics for evaluating parameter identifiability and error reduction
journal, March 2009


Technical note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models
journal, January 2013


A short exploration of structural noise: A SHORT EXPLORATION OF STRUCTURAL NOISE
journal, May 2010


Pursuing the method of multiple working hypotheses for hydrological modeling: HYPOTHESIS TESTING IN HYDROLOGY
journal, September 2011


Quantifying Data Worth Toward Reducing Predictive Uncertainty
journal, August 2010


Response to Comment on “Two statistics for evaluating parameter identifiability and error reduction”
journal, January 2010


Review of surrogate modeling in water resources: REVIEW
journal, July 2012


Simulation of Water-Table Aquifers Using Specified Saturated Thickness
journal, February 2014


Ignorance is bliss: Or seven reasons not to use uncertainty analysis: OPINION
journal, May 2006


Decision tree for choosing an uncertainty analysis methodology: a wiki experiment
journal, January 2006


Towards a comprehensive assessment of model structural adequacy: ASSESSMENT OF MODEL STRUCTURAL ADEQUACY
journal, August 2012


Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences
journal, February 1994


Beyond optimality: Multistakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty
journal, October 2014


Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment
journal, January 2005


Bees, lies and evidence-based policy
journal, February 2013


Probabilistic sensitivity analysis of complex models: a Bayesian approach
journal, August 2004


Are Models Too Simple? Arguments for Increased Parameterization
journal, May 2007


Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis
journal, January 2013


Evolutionary multiobjective optimization in water resources: The past, present, and future
journal, January 2013


Nonlinear sensitivity analysis of multiparameter model systems
journal, January 1978


Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation
journal, January 2007


Sensitivity analysis of simulation models: an overview
journal, January 2010


A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output
journal, February 1999


Knowledge, transparency, and refutability in groundwater models, an example from the Death Valley regional groundwater flow system
journal, January 2013


Reducing the computational cost of automatic calibration through model preemption: MODEL PREEMPTION APPROACH IN AUTOMATIC CALIBRATION
journal, November 2010


A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY
journal, October 2005


Two-Dimensional Advective Transport in Ground-Water Flow Parameter Estimation
journal, November 1996


Effective and efficient global optimization for conceptual rainfall-runoff models
journal, April 1992


The Logic of Scientific Discovery
journal, November 1959


Works referencing / citing this record:

Signature-Domain Calibration of Hydrological Models Using Approximate Bayesian Computation: Empirical Analysis of Fundamental Properties
journal, June 2018


Why Should Practitioners be Concerned about Predictive Uncertainty of Groundwater Management Models?
journal, October 2016


Terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution
journal, January 2019


A Bayesian approach to improved calibration and prediction of groundwater models with structural error
journal, November 2015


The evolution of process-based hydrologic models: historical challenges and the collective quest for physical realism
journal, January 2017


The Fast and the Robust: Trade‐Offs Between Optimization Robustness and Cost in the Calibration of Environmental Models
journal, November 2018


Characterizing and reducing equifinality by constraining a distributed catchment model with regional signatures, local observations, and process understanding
journal, January 2017


Model Variable Augmentation (MVA) for Diagnostic Assessment of Sensitivity Analysis Results
journal, April 2019


Comparison of Newton-type and SCE optimisation algorithms for the calibration of conceptual hydrological models
journal, July 2016


A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: Benchmarking Against Industry‐Standard Algorithms
journal, November 2018


A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: From Standard Gauss‐Newton to Robust Gauss‐Newton
journal, November 2018