Practical Use of Computationally Frugal Model Analysis Methods
Abstract
Abstract Three challenges compromise the utility of mathematical models of groundwater and other environmental systems: (1) a dizzying array of model analysis methods and metrics make it difficult to compare evaluations of model adequacy, sensitivity, and uncertainty; (2) the high computational demands of many popular model analysis methods (requiring 1000's, 10,000 s, or more model runs) make them difficult to apply to complex models; and (3) many models are plagued by unrealistic nonlinearities arising from the numerical model formulation and implementation. This study proposes a strategy to address these challenges through a careful combination of model analysis and implementation methods. In this strategy, computationally frugal model analysis methods (often requiring a few dozen parallelizable model runs) play a major role, and computationally demanding methods are used for problems where (relatively) inexpensive diagnostics suggest the frugal methods are unreliable. We also argue in favor of detecting and, where possible, eliminating unrealistic model nonlinearities—this increases the realism of the model itself and facilitates the application of frugal methods. Literature examples are used to demonstrate the use of frugal methods and associated diagnostics. We suggest that the strategy proposed in this paper would allow the environmental sciences community to achieve greater transparency and falsifiabilitymore »
- Authors:
-
- U.S. Geological Survey, Boulder, CO (United States); Univ. of Kansas, Lawrence, KS (United States)
- Univ. of Adelaide, SA (Australia)
- National Center for Atmospheric Research, Boulder, CO (United States)
- Florida State Univ., Tallahassee, FL (United States)
- Colorado State Univ., Fort Collins, CO (United States)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Univ. of Darmstadt (Germany)
- California State Univ. (CalState), Chico, CA (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE Laboratory Directed Research and Development (LDRD) Program
- OSTI Identifier:
- 1286771
- Alternate Identifier(s):
- OSTI ID: 1400448
- Grant/Contract Number:
- AC05-00OR22725; SC0008272; 0911074; 21-66885
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Ground Water
- Additional Journal Information:
- Journal Volume: 54; Journal Issue: 2; Journal ID: ISSN 0017-467X
- Publisher:
- Wiley - NGWA
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 54 ENVIRONMENTAL SCIENCES; Model calibration; Sensitivity analysis; Uncertainty; Bayesian; regression; transparency; falsifiability
Citation Formats
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States: N. p., 2015.
Web. doi:10.1111/gwat.12330.
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, & Mehl, Steffen. Practical Use of Computationally Frugal Model Analysis Methods. United States. https://doi.org/10.1111/gwat.12330
Hill, Mary C., Kavetski, Dmitri, Clark, Martyn, Ye, Ming, Arabi, Mazdak, Lu, Dan, Foglia, Laura, and Mehl, Steffen. Sat .
"Practical Use of Computationally Frugal Model Analysis Methods". United States. https://doi.org/10.1111/gwat.12330. https://www.osti.gov/servlets/purl/1286771.
@article{osti_1286771,
title = {Practical Use of Computationally Frugal Model Analysis Methods},
author = {Hill, Mary C. and Kavetski, Dmitri and Clark, Martyn and Ye, Ming and Arabi, Mazdak and Lu, Dan and Foglia, Laura and Mehl, Steffen},
abstractNote = {Abstract Three challenges compromise the utility of mathematical models of groundwater and other environmental systems: (1) a dizzying array of model analysis methods and metrics make it difficult to compare evaluations of model adequacy, sensitivity, and uncertainty; (2) the high computational demands of many popular model analysis methods (requiring 1000's, 10,000 s, or more model runs) make them difficult to apply to complex models; and (3) many models are plagued by unrealistic nonlinearities arising from the numerical model formulation and implementation. This study proposes a strategy to address these challenges through a careful combination of model analysis and implementation methods. In this strategy, computationally frugal model analysis methods (often requiring a few dozen parallelizable model runs) play a major role, and computationally demanding methods are used for problems where (relatively) inexpensive diagnostics suggest the frugal methods are unreliable. We also argue in favor of detecting and, where possible, eliminating unrealistic model nonlinearities—this increases the realism of the model itself and facilitates the application of frugal methods. Literature examples are used to demonstrate the use of frugal methods and associated diagnostics. We suggest that the strategy proposed in this paper would allow the environmental sciences community to achieve greater transparency and falsifiability of environmental models, and obtain greater scientific insight from ongoing and future modeling efforts.},
doi = {10.1111/gwat.12330},
journal = {Ground Water},
number = 2,
volume = 54,
place = {United States},
year = {Sat Mar 21 00:00:00 EDT 2015},
month = {Sat Mar 21 00:00:00 EDT 2015}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 48 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Exploring Regression Structure Using Nonparametric Functional Estimation
journal, September 1993
- Samarov, Alexander M.
- Journal of the American Statistical Association, Vol. 88, Issue 423
Uncertainty in science and its role in climate policy
journal, December 2011
- Smith, Leonard A.; Stern, Nicholas
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 369, Issue 1956
Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water: OPINION
journal, May 2011
- Wood, Eric F.; Roundy, Joshua K.; Troy, Tara J.
- Water Resources Research, Vol. 47, Issue 5
Testing Alternative Ground Water Models Using Cross-Validation and Other Methods
journal, September 2007
- Foglia, L.; Mehl, S. W.; Hill, M. C.
- Ground Water, Vol. 45, Issue 5
Two statistics for evaluating parameter identifiability and error reduction
journal, March 2009
- Doherty, John; Hunt, Randall J.
- Journal of Hydrology, Vol. 366, Issue 1-4
Technical note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models
journal, January 2013
- Herman, J. D.; Kollat, J. B.; Reed, P. M.
- Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 4
A short exploration of structural noise: A SHORT EXPLORATION OF STRUCTURAL NOISE
journal, May 2010
- Doherty, John; Welter, David
- Water Resources Research, Vol. 46, Issue 5
Comment on “Two statistics for evaluating parameter identifiability and error reduction” by John Doherty and Randall J. Hunt
journal, January 2010
- Hill, Mary C.
- Journal of Hydrology, Vol. 380, Issue 3-4
Pursuing the method of multiple working hypotheses for hydrological modeling: HYPOTHESIS TESTING IN HYDROLOGY
journal, September 2011
- Clark, Martyn P.; Kavetski, Dmitri; Fenicia, Fabrizio
- Water Resources Research, Vol. 47, Issue 9
Quantifying Data Worth Toward Reducing Predictive Uncertainty
journal, August 2010
- Dausman, Alyssa M.; Doherty, John; Langevin, Christian D.
- Ground Water, Vol. 48, Issue 5
Response to Comment on “Two statistics for evaluating parameter identifiability and error reduction”
journal, January 2010
- Doherty, John; Hunt, Randall J.
- Journal of Hydrology, Vol. 380, Issue 3-4
Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction
journal, October 2010
- Kavetski, Dmitri; Clark, Martyn P.
- Water Resources Research, Vol. 46, Issue 10
Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches
journal, October 2006
- Borgonovo, Emanuele
- Risk Analysis, Vol. 26, Issue 5
Effective Groundwater Model Calibration
book, January 2007
- Hill, Mary C.; Tiedeman, Claire R.
- John Wiley & Sons, Inc.
Review of surrogate modeling in water resources: REVIEW
journal, July 2012
- Razavi, Saman; Tolson, Bryan A.; Burn, Donald H.
- Water Resources Research, Vol. 48, Issue 7
Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information
journal, February 1986
- Carrera, Jesus; Neuman, Shlomo P.
- Water Resources Research, Vol. 22, Issue 2
Simulation of Water-Table Aquifers Using Specified Saturated Thickness
journal, February 2014
- Sheets, Rodney A.; Hill, Mary C.; Haitjema, Henk M.
- Groundwater, Vol. 53, Issue 1
Analysis of regression confidence intervals and Bayesian credible intervals for uncertainty quantification: REGRESSION CONFIDENCE AND BAYESIAN CREDIBLE INTERVALS
journal, September 2012
- Lu, Dan; Ye, Ming; Hill, Mary C.
- Water Resources Research, Vol. 48, Issue 9
Parameter and observation importance in modelling virus transport in saturated porous media—investigations in a homogenous system
journal, November 2005
- Barth, Gilbert R.; Hill, Mary C.
- Journal of Contaminant Hydrology, Vol. 80, Issue 3-4
Ignorance is bliss: Or seven reasons not to use uncertainty analysis: OPINION
journal, May 2006
- Pappenberger, F.; Beven, K. J.
- Water Resources Research, Vol. 42, Issue 5
Decision tree for choosing an uncertainty analysis methodology: a wiki experiment
journal, January 2006
- Pappenberger, Florian; Harvey, Hamish; Beven, Keith
- Hydrological Processes, Vol. 20, Issue 17
Towards a comprehensive assessment of model structural adequacy: ASSESSMENT OF MODEL STRUCTURAL ADEQUACY
journal, August 2012
- Gupta, Hoshin V.; Clark, Martyn P.; Vrugt, Jasper A.
- Water Resources Research, Vol. 48, Issue 8
Sensitivity analysis, calibration, and testing of a distributed hydrological model using error-based weighting and one objective function: HYDROLOGICAL MODEL CALIBRATION
journal, June 2009
- Foglia, L.; Hill, M. C.; Mehl, S. W.
- Water Resources Research, Vol. 45, Issue 6
Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences
journal, February 1994
- Oreskes, N.; Shrader-Frechette, K.; Belitz, K.
- Science, Vol. 263, Issue 5147
Beyond optimality: Multistakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty
journal, October 2014
- Herman, Jonathan D.; Zeff, Harrison B.; Reed, Patrick M.
- Water Resources Research, Vol. 50, Issue 10
Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment
journal, January 2005
- Liu, Z.; Martina, M. L. V.; Todini, E.
- Hydrology and Earth System Sciences, Vol. 9, Issue 4
Evaluating model structure adequacy: The case of the Maggia Valley groundwater system, southern Switzerland: Evaluating Model Structure Adequacy
journal, January 2013
- Foglia, L.; Mehl, S. W.; Hill, M. C.
- Water Resources Research, Vol. 49, Issue 1
Bees, lies and evidence-based policy
journal, February 2013
- Dicks, Lynn
- Nature, Vol. 494, Issue 7437
Assessment and management of risk in subsurface hydrology: A review and perspective
journal, January 2013
- Tartakovsky, Daniel M.
- Advances in Water Resources, Vol. 51
Probabilistic sensitivity analysis of complex models: a Bayesian approach
journal, August 2004
- Oakley, Jeremy E.; O'Hagan, Anthony
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3
Are Models Too Simple? Arguments for Increased Parameterization
journal, May 2007
- Hunt, Randall J.; Doherty, John; Tonkin, Matthew J.
- Ground Water, Vol. 45, Issue 3
Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis
journal, January 2013
- Li, J. D.; Duan, Q. Y.; Gong, W.
- Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 2
Evolutionary multiobjective optimization in water resources: The past, present, and future
journal, January 2013
- Reed, P. M.; Hadka, D.; Herman, J. D.
- Advances in Water Resources, Vol. 51
Methods for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional groundwater flow system: USING MODEL PREDICTIONS TO GUIDE DATA COLLECTION
journal, January 2003
- Tiedeman, Claire R.; Hill, Mary C.; D'Agnese, Frank A.
- Water Resources Research, Vol. 39, Issue 1
Nonlinear sensitivity analysis of multiparameter model systems
journal, January 1978
- Cukier, R. I.; Levine, H. B.; Shuler, K. E.
- Journal of Computational Physics, Vol. 26, Issue 1
Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation
journal, January 2007
- Tang, Y.; Reed, P.; Wagener, T.
- Hydrology and Earth System Sciences, Vol. 11, Issue 2
Sensitivity analysis of simulation models: an overview
journal, January 2010
- Kleijnen, Jack P. C.
- Procedia - Social and Behavioral Sciences, Vol. 2, Issue 6
A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output
journal, February 1999
- Saltelli, A.; Tarantola, S.; Chan, K. P. -S.
- Technometrics, Vol. 41, Issue 1
Knowledge, transparency, and refutability in groundwater models, an example from the Death Valley regional groundwater flow system
journal, January 2013
- Hill, Mary C.; Faunt, Claudia C.; Belcher, Wayne R.
- Physics and Chemistry of the Earth, Parts A/B/C, Vol. 64
Model smoothing strategies to remove microscale discontinuities and spurious secondary optima in objective functions in hydrological calibration: MODEL SMOOTHING IN HYDROLOGY
journal, March 2007
- Kavetski, Dmitri; Kuczera, George
- Water Resources Research, Vol. 43, Issue 3
Reducing the computational cost of automatic calibration through model preemption: MODEL PREEMPTION APPROACH IN AUTOMATIC CALIBRATION
journal, November 2010
- Razavi, Saman; Tolson, Bryan A.; Matott, L. Shawn
- Water Resources Research, Vol. 46, Issue 11
Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models: Distributed evaluation of local sensitivity analysis
journal, January 2014
- Rakovec, O.; Hill, M. C.; Clark, M. P.
- Water Resources Research, Vol. 50, Issue 1
A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY
journal, October 2005
- Tonkin, Matthew James; Doherty, John
- Water Resources Research, Vol. 41, Issue 10
Two-Dimensional Advective Transport in Ground-Water Flow Parameter Estimation
journal, November 1996
- Anderman, Evan R.; Hill, Mary C.; Poeter, Eileen P.
- Ground Water, Vol. 34, Issue 6
Effective and efficient global optimization for conceptual rainfall-runoff models
journal, April 1992
- Duan, Qingyun; Sorooshian, Soroosh; Gupta, Vijai
- Water Resources Research, Vol. 28, Issue 4
Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior: TIME-VARYING SENSITIVITY OF WATERSHED MODELS
journal, March 2013
- Herman, J. D.; Reed, P. M.; Wagener, T.
- Water Resources Research, Vol. 49, Issue 3
The Logic of Scientific Discovery
journal, November 1959
- Popper, Karl R.; Weiss, George
- Physics Today, Vol. 12, Issue 11
OPR-PPR, a computer program for assessing data importance to model predictions using linear statistics
report, January 2007
- Tonkin, Matthew J.; Tiedeman, Claire; Ely, D. Matthew
Works referencing / citing this record:
Why Should Practitioners be Concerned about Predictive Uncertainty of Groundwater Management Models?
journal, October 2016
- Delottier, H.; Pryet, A.; Dupuy, A.
- Water Resources Management, Vol. 31, Issue 1
Terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution
journal, January 2019
- Barnhart, Katherine R.; Glade, Rachel C.; Shobe, Charles M.
- Geoscientific Model Development, Vol. 12, Issue 4
Reconnecting Stochastic Methods With Hydrogeological Applications: A Utilitarian Uncertainty Analysis and Risk Assessment Approach for the Design of Optimal Monitoring Networks
journal, March 2018
- Bode, Felix; Ferré, Ty; Zigelli, Niklas
- Water Resources Research, Vol. 54, Issue 3
The evolution of process-based hydrologic models: historical challenges and the collective quest for physical realism
journal, January 2017
- Clark, Martyn P.; Bierkens, Marc F. P.; Samaniego, Luis
- Hydrology and Earth System Sciences, Vol. 21, Issue 7
Characterizing and reducing equifinality by constraining a distributed catchment model with regional signatures, local observations, and process understanding
journal, January 2017
- Kelleher, Christa; McGlynn, Brian; Wagener, Thorsten
- Hydrology and Earth System Sciences, Vol. 21, Issue 7
Sensitivity Analysis and Calibration of an Integrated Hydrologic Model in an Irrigated Agricultural Basin With a Groundwater‐Dependent Ecosystem
journal, September 2019
- Tolley, D.; Foglia, L.; Harter, T.
- Water Resources Research, Vol. 55, Issue 9
Comparison of Newton-type and SCE optimisation algorithms for the calibration of conceptual hydrological models
journal, July 2016
- Qin, Youwei; Kuczera, George; Kavetski, Dmitri
- Australasian Journal of Water Resources, Vol. 20, Issue 2
A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: From Standard Gauss‐Newton to Robust Gauss‐Newton
journal, November 2018
- Qin, Youwei; Kavetski, Dmitri; Kuczera, George
- Water Resources Research, Vol. 54, Issue 11
terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution
posted_content, October 2018
- Barnhart, Katherine R.; Glade, Rachel C.; Shobe, Charles M.
- Geoscientific Model Development