skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Practical Use of Computationally Frugal Model Analysis Methods

Abstract

Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts

Authors:
 [1];  [2];  [3];  [4];  [5];  [6];  [7];  [8]
  1. U.S. Geological Survey, Boulder, CO (United States); Univ. of Kansas, Lawrence, KS (United States)
  2. Univ. of Adelaide, SA (Australia)
  3. National Center for Atmospheric Research, Boulder, CO (United States)
  4. Florida State Univ., Tallahassee, FL (United States)
  5. Colorado State Univ., Fort Collins, CO (United States)
  6. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  7. Univ. of Darmstadt (Germany)
  8. California State Univ. (CalState), Chico, CA (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1286771
Alternate Identifier(s):
OSTI ID: 1400448
Grant/Contract Number:  
AC05-00OR22725; SC0008272; 0911074; 21-66885
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Ground Water
Additional Journal Information:
Journal Volume: 54; Journal Issue: 2; Journal ID: ISSN 0017-467X
Publisher:
Wiley - NGWA
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; 54 ENVIRONMENTAL SCIENCES; Model calibration; Sensitivity analysis; Uncertainty; Bayesian; regression; transparency; falsifiability

Citation Formats

Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States: N. p., 2015. Web. doi:10.1111/gwat.12330.
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, & Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States. doi:10.1111/gwat.12330.
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. Sat . "Practical Use of Computationally Frugal Model Analysis Methods". United States. doi:10.1111/gwat.12330. https://www.osti.gov/servlets/purl/1286771.
@article{osti_1286771,
title = {Practical Use of Computationally Frugal Model Analysis Methods},
author = {Hill, Mary C. and Kavetski, Dmitri and Clark, Martyn and Ye, Ming and Arabi, Mazdak and Lu, Dan and Foglia, Laura and Mehl, Steffen},
abstractNote = {Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts},
doi = {10.1111/gwat.12330},
journal = {Ground Water},
issn = {0017-467X},
number = 2,
volume = 54,
place = {United States},
year = {2015},
month = {3}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 13 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Exploring Regression Structure Using Nonparametric Functional Estimation
journal, September 1993


Uncertainty in science and its role in climate policy
journal, December 2011

  • Smith, Leonard A.; Stern, Nicholas
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 369, Issue 1956
  • DOI: 10.1098/rsta.2011.0149

Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water: OPINION
journal, May 2011

  • Wood, Eric F.; Roundy, Joshua K.; Troy, Tara J.
  • Water Resources Research, Vol. 47, Issue 5
  • DOI: 10.1029/2010WR010090

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences
journal, February 1994


Testing Alternative Ground Water Models Using Cross-Validation and Other Methods
journal, September 2007


Two statistics for evaluating parameter identifiability and error reduction
journal, March 2009


Beyond optimality: Multistakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty
journal, October 2014

  • Herman, Jonathan D.; Zeff, Harrison B.; Reed, Patrick M.
  • Water Resources Research, Vol. 50, Issue 10
  • DOI: 10.1002/2014WR015338

Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment
journal, January 2005

  • Liu, Z.; Martina, M. L. V.; Todini, E.
  • Hydrology and Earth System Sciences, Vol. 9, Issue 4
  • DOI: 10.5194/hess-9-347-2005

Evaluating model structure adequacy: The case of the Maggia Valley groundwater system, southern Switzerland: Evaluating Model Structure Adequacy
journal, January 2013

  • Foglia, L.; Mehl, S. W.; Hill, M. C.
  • Water Resources Research, Vol. 49, Issue 1
  • DOI: 10.1029/2011WR011779

Technical note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models
journal, January 2013

  • Herman, J. D.; Kollat, J. B.; Reed, P. M.
  • Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 4
  • DOI: 10.5194/hessd-10-4275-2013

Bees, lies and evidence-based policy
journal, February 2013


Assessment and management of risk in subsurface hydrology: A review and perspective
journal, January 2013


Probabilistic sensitivity analysis of complex models: a Bayesian approach
journal, August 2004

  • Oakley, Jeremy E.; O'Hagan, Anthony
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3
  • DOI: 10.1111/j.1467-9868.2004.05304.x

A short exploration of structural noise: A SHORT EXPLORATION OF STRUCTURAL NOISE
journal, May 2010


Pursuing the method of multiple working hypotheses for hydrological modeling: HYPOTHESIS TESTING IN HYDROLOGY
journal, September 2011

  • Clark, Martyn P.; Kavetski, Dmitri; Fenicia, Fabrizio
  • Water Resources Research, Vol. 47, Issue 9
  • DOI: 10.1029/2010WR009827

Quantifying Data Worth Toward Reducing Predictive Uncertainty
journal, August 2010


Response to Comment on “Two statistics for evaluating parameter identifiability and error reduction”
journal, January 2010


Are Models Too Simple? Arguments for Increased Parameterization
journal, May 2007


Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction
journal, October 2010

  • Kavetski, Dmitri; Clark, Martyn P.
  • Water Resources Research, Vol. 46, Issue 10
  • DOI: 10.1029/2009WR008896

Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches
journal, October 2006


Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis
journal, January 2013

  • Li, J. D.; Duan, Q. Y.; Gong, W.
  • Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 2
  • DOI: 10.5194/hessd-10-2243-2013

Evolutionary multiobjective optimization in water resources: The past, present, and future
journal, January 2013


Review of surrogate modeling in water resources: REVIEW
journal, July 2012

  • Razavi, Saman; Tolson, Bryan A.; Burn, Donald H.
  • Water Resources Research, Vol. 48, Issue 7
  • DOI: 10.1029/2011WR011527

Nonlinear sensitivity analysis of multiparameter model systems
journal, January 1978


Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information
journal, February 1986


Simulation of Water-Table Aquifers Using Specified Saturated Thickness
journal, February 2014

  • Sheets, Rodney A.; Hill, Mary C.; Haitjema, Henk M.
  • Groundwater, Vol. 53, Issue 1
  • DOI: 10.1111/gwat.12164

Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation
journal, January 2007


Sensitivity analysis of simulation models: an overview
journal, January 2010


Parameter and observation importance in modelling virus transport in saturated porous media—investigations in a homogenous system
journal, November 2005


A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output
journal, February 1999


Knowledge, transparency, and refutability in groundwater models, an example from the Death Valley regional groundwater flow system
journal, January 2013

  • Hill, Mary C.; Faunt, Claudia C.; Belcher, Wayne R.
  • Physics and Chemistry of the Earth, Parts A/B/C, Vol. 64
  • DOI: 10.1016/j.pce.2013.03.006

Reducing the computational cost of automatic calibration through model preemption: MODEL PREEMPTION APPROACH IN AUTOMATIC CALIBRATION
journal, November 2010

  • Razavi, Saman; Tolson, Bryan A.; Matott, L. Shawn
  • Water Resources Research, Vol. 46, Issue 11
  • DOI: 10.1029/2009WR008957

Ignorance is bliss: Or seven reasons not to use uncertainty analysis: OPINION
journal, May 2006

  • Pappenberger, F.; Beven, K. J.
  • Water Resources Research, Vol. 42, Issue 5
  • DOI: 10.1029/2005WR004820

A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY
journal, October 2005

  • Tonkin, Matthew James; Doherty, John
  • Water Resources Research, Vol. 41, Issue 10
  • DOI: 10.1029/2005WR003995

Two-Dimensional Advective Transport in Ground-Water Flow Parameter Estimation
journal, November 1996


Decision tree for choosing an uncertainty analysis methodology: a wiki experiment
journal, January 2006

  • Pappenberger, Florian; Harvey, Hamish; Beven, Keith
  • Hydrological Processes, Vol. 20, Issue 17
  • DOI: 10.1002/hyp.6541

Towards a comprehensive assessment of model structural adequacy: ASSESSMENT OF MODEL STRUCTURAL ADEQUACY
journal, August 2012

  • Gupta, Hoshin V.; Clark, Martyn P.; Vrugt, Jasper A.
  • Water Resources Research, Vol. 48, Issue 8
  • DOI: 10.1029/2011WR011044

Effective and efficient global optimization for conceptual rainfall-runoff models
journal, April 1992

  • Duan, Qingyun; Sorooshian, Soroosh; Gupta, Vijai
  • Water Resources Research, Vol. 28, Issue 4
  • DOI: 10.1029/91WR02985

Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior: TIME-VARYING SENSITIVITY OF WATERSHED MODELS
journal, March 2013

  • Herman, J. D.; Reed, P. M.; Wagener, T.
  • Water Resources Research, Vol. 49, Issue 3
  • DOI: 10.1002/wrcr.20124