Practical Use of Computationally Frugal Model Analysis Methods

Hill, Mary C.; Kavetski, Dmitri; Clark, Martyn; Ye, Ming; Arabi, Mazdak; Lu, Dan; Foglia, Laura; Mehl, Steffen

doi:10.1111/gwat.12330

Practical Use of Computationally Frugal Model Analysis Methods

Journal Article · Sat Mar 21 00:00:00 EDT 2015 · Ground Water

DOI:https://doi.org/10.1111/gwat.12330· OSTI ID:1286771

Hill, Mary C. ^[1]; Kavetski, Dmitri ^[2]; Clark, Martyn ^[3]; Ye, Ming ^[4]; Arabi, Mazdak ^[5]; Lu, Dan ^[6]; Foglia, Laura ^[7]; Mehl, Steffen ^[8]

U.S. Geological Survey, Boulder, CO (United States); Univ. of Kansas, Lawrence, KS (United States)
Univ. of Adelaide, SA (Australia)
National Center for Atmospheric Research, Boulder, CO (United States)
Florida State Univ., Tallahassee, FL (United States)
Colorado State Univ., Fort Collins, CO (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Univ. of Darmstadt (Germany)
California State Univ. (CalState), Chico, CA (United States)

Computationally frugal methods of model analysis can provide substantial benefits when developing models of groundwater and other environmental systems. Model analysis includes ways to evaluate model adequacy and to perform sensitivity and uncertainty analysis. Frugal methods typically require 10s of parallelizable model runs; their convenience allows for other uses of the computational effort. We suggest that model analysis be posed as a set of questions used to organize methods that range from frugal to expensive (requiring 10,000 model runs or more). This encourages focus on method utility, even when methods have starkly different theoretical backgrounds. We note that many frugal methods are more useful when unrealistic process-model nonlinearities are reduced. Inexpensive diagnostics are identified for determining when frugal methods are advantageous. Examples from the literature are used to demonstrate local methods and the diagnostics. We suggest that the greater use of computationally frugal model analysis methods would allow questions such as those posed in this work to be addressed more routinely, allowing the environmental sciences community to obtain greater scientific insight from the many ongoing and future modeling efforts

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: ORNL LDRD Director's R&D; USDOE

Grant/Contract Number:: AC05-00OR22725; SC0008272

OSTI ID:: 1286771

Alternate ID(s):: OSTI ID: 1400448

Journal Information:: Ground Water, Journal Name: Ground Water Journal Issue: 2 Vol. 54; ISSN 0017-467X

Publisher:: Wiley - NGWACopyright Statement

Country of Publication:: United States

Language:: English

References (49)

Effective Groundwater Model Calibration Hill, Mary C.; Tiedeman, Claire R. John Wiley & Sons, Inc. https://doi.org/10.1002/0470041080	book	January 2007
Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models: Distributed evaluation of local sensitivity analysis Rakovec, O.; Hill, M. C.; Clark, M. P. Water Resources Research, Vol. 50, Issue 1 https://doi.org/10.1002/2013WR014063	journal	January 2014
Beyond optimality: Multistakeholder robustness tradeoffs for regional water portfolio planning under deep uncertainty Herman, Jonathan D.; Zeff, Harrison B.; Reed, Patrick M. Water Resources Research, Vol. 50, Issue 10 https://doi.org/10.1002/2014WR015338	journal	October 2014
Applied Regression Analysis Draper, Norman R.; Smith, Harry https://doi.org/10.1002/9781118625590	book	January 1998
Decision tree for choosing an uncertainty analysis methodology: a wiki experiment Pappenberger, Florian; Harvey, Hamish; Beven, Keith Hydrological Processes, Vol. 20, Issue 17 https://doi.org/10.1002/hyp.6541	journal	January 2006
Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior: TIME-VARYING SENSITIVITY OF WATERSHED MODELS Herman, J. D.; Reed, P. M.; Wagener, T. Water Resources Research, Vol. 49, Issue 3 https://doi.org/10.1002/wrcr.20124	journal	March 2013
Nonlinear sensitivity analysis of multiparameter model systems Cukier, R. I.; Levine, H. B.; Shuler, K. E. Journal of Computational Physics, Vol. 26, Issue 1 https://doi.org/10.1016/0021-9991(78)90097-9	journal	January 1978
Evolutionary multiobjective optimization in water resources: The past, present, and future Reed, P. M.; Hadka, D.; Herman, J. D. Advances in Water Resources, Vol. 51 https://doi.org/10.1016/j.advwatres.2012.01.005	journal	January 2013
Assessment and management of risk in subsurface hydrology: A review and perspective Tartakovsky, Daniel M. Advances in Water Resources, Vol. 51 https://doi.org/10.1016/j.advwatres.2012.04.007	journal	January 2013
Parameter and observation importance in modelling virus transport in saturated porous media—investigations in a homogenous system Barth, Gilbert R.; Hill, Mary C. Journal of Contaminant Hydrology, Vol. 80, Issue 3-4 https://doi.org/10.1016/j.jconhyd.2005.06.012	journal	November 2005
Two statistics for evaluating parameter identifiability and error reduction Doherty, John; Hunt, Randall J. Journal of Hydrology, Vol. 366, Issue 1-4 https://doi.org/10.1016/j.jhydrol.2008.12.018	journal	March 2009
Comment on “Two statistics for evaluating parameter identifiability and error reduction” by John Doherty and Randall J. Hunt Hill, Mary C. Journal of Hydrology, Vol. 380, Issue 3-4 https://doi.org/10.1016/j.jhydrol.2009.10.011	journal	January 2010
Response to Comment on “Two statistics for evaluating parameter identifiability and error reduction” Doherty, John; Hunt, Randall J. Journal of Hydrology, Vol. 380, Issue 3-4 https://doi.org/10.1016/j.jhydrol.2009.10.012	journal	January 2010
Knowledge, transparency, and refutability in groundwater models, an example from the Death Valley regional groundwater flow system Hill, Mary C.; Faunt, Claudia C.; Belcher, Wayne R. Physics and Chemistry of the Earth, Parts A/B/C, Vol. 64 https://doi.org/10.1016/j.pce.2013.03.006	journal	January 2013
Sensitivity analysis of simulation models: an overview Kleijnen, Jack P. C. Procedia - Social and Behavioral Sciences, Vol. 2, Issue 6 https://doi.org/10.1016/j.sbspro.2010.05.130	journal	January 2010
Methods for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional groundwater flow system: USING MODEL PREDICTIONS TO GUIDE DATA COLLECTION Tiedeman, Claire R.; Hill, Mary C.; D'Agnese, Frank A. Water Resources Research, Vol. 39, Issue 1 https://doi.org/10.1029/2001WR001255	journal	January 2003
A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY Tonkin, Matthew James; Doherty, John Water Resources Research, Vol. 41, Issue 10 https://doi.org/10.1029/2005WR003995	journal	October 2005
Ignorance is bliss: Or seven reasons not to use uncertainty analysis: OPINION Pappenberger, F.; Beven, K. J. Water Resources Research, Vol. 42, Issue 5 https://doi.org/10.1029/2005WR004820	journal	May 2006
Model smoothing strategies to remove microscale discontinuities and spurious secondary optima in objective functions in hydrological calibration: MODEL SMOOTHING IN HYDROLOGY Kavetski, Dmitri; Kuczera, George Water Resources Research, Vol. 43, Issue 3 https://doi.org/10.1029/2006WR005195	journal	March 2007
Sensitivity analysis, calibration, and testing of a distributed hydrological model using error-based weighting and one objective function: HYDROLOGICAL MODEL CALIBRATION Foglia, L.; Hill, M. C.; Mehl, S. W. Water Resources Research, Vol. 45, Issue 6 https://doi.org/10.1029/2008WR007255	journal	June 2009
A short exploration of structural noise: A SHORT EXPLORATION OF STRUCTURAL NOISE Doherty, John; Welter, David Water Resources Research, Vol. 46, Issue 5 https://doi.org/10.1029/2009WR008377	journal	May 2010
Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction Kavetski, Dmitri; Clark, Martyn P. Water Resources Research, Vol. 46, Issue 10 https://doi.org/10.1029/2009WR008896	journal	October 2010
Reducing the computational cost of automatic calibration through model preemption: MODEL PREEMPTION APPROACH IN AUTOMATIC CALIBRATION Razavi, Saman; Tolson, Bryan A.; Matott, L. Shawn Water Resources Research, Vol. 46, Issue 11 https://doi.org/10.1029/2009WR008957	journal	November 2010
Pursuing the method of multiple working hypotheses for hydrological modeling: HYPOTHESIS TESTING IN HYDROLOGY Clark, Martyn P.; Kavetski, Dmitri; Fenicia, Fabrizio Water Resources Research, Vol. 47, Issue 9 https://doi.org/10.1029/2010WR009827	journal	September 2011
Hyperresolution global land surface modeling: Meeting a grand challenge for monitoring Earth's terrestrial water: OPINION Wood, Eric F.; Roundy, Joshua K.; Troy, Tara J. Water Resources Research, Vol. 47, Issue 5 https://doi.org/10.1029/2010WR010090	journal	May 2011
Towards a comprehensive assessment of model structural adequacy: ASSESSMENT OF MODEL STRUCTURAL ADEQUACY Gupta, Hoshin V.; Clark, Martyn P.; Vrugt, Jasper A. Water Resources Research, Vol. 48, Issue 8 https://doi.org/10.1029/2011WR011044	journal	August 2012
Analysis of regression confidence intervals and Bayesian credible intervals for uncertainty quantification: REGRESSION CONFIDENCE AND BAYESIAN CREDIBLE INTERVALS Lu, Dan; Ye, Ming; Hill, Mary C. Water Resources Research, Vol. 48, Issue 9 https://doi.org/10.1029/2011WR011289	journal	September 2012
Review of surrogate modeling in water resources: REVIEW Razavi, Saman; Tolson, Bryan A.; Burn, Donald H. Water Resources Research, Vol. 48, Issue 7 https://doi.org/10.1029/2011WR011527	journal	July 2012
Evaluating model structure adequacy: The case of the Maggia Valley groundwater system, southern Switzerland: Evaluating Model Structure Adequacy Foglia, L.; Mehl, S. W.; Hill, M. C. Water Resources Research, Vol. 49, Issue 1 https://doi.org/10.1029/2011WR011779	journal	January 2013
Effective and efficient global optimization for conceptual rainfall-runoff models Duan, Qingyun; Sorooshian, Soroosh; Gupta, Vijai Water Resources Research, Vol. 28, Issue 4 https://doi.org/10.1029/91WR02985	journal	April 1992
Estimation of Aquifer Parameters Under Transient and Steady State Conditions: 1. Maximum Likelihood Method Incorporating Prior Information Carrera, Jesus; Neuman, Shlomo P. Water Resources Research, Vol. 22, Issue 2 https://doi.org/10.1029/WR022i002p00199	journal	February 1986
Bees, lies and evidence-based policy Dicks, Lynn Nature, Vol. 494, Issue 7437 https://doi.org/10.1038/494283a	journal	February 2013
The Logic of Scientific Discovery Popper, Karl R.; Weiss, George Physics Today, Vol. 12, Issue 11 https://doi.org/10.1063/1.3060577	journal	November 1959
A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output Saltelli, A.; Tarantola, S.; Chan, K. P. -S. Technometrics, Vol. 41, Issue 1 https://doi.org/10.1080/00401706.1999.10485594	journal	February 1999
Exploring Regression Structure Using Nonparametric Functional Estimation Samarov, Alexander M. Journal of the American Statistical Association, Vol. 88, Issue 423 https://doi.org/10.1080/01621459.1993.10476348	journal	September 1993
Uncertainty in science and its role in climate policy Smith, Leonard A.; Stern, Nicholas Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 369, Issue 1956 https://doi.org/10.1098/rsta.2011.0149	journal	December 2011
Simulation of Water-Table Aquifers Using Specified Saturated Thickness Sheets, Rodney A.; Hill, Mary C.; Haitjema, Henk M. Groundwater, Vol. 53, Issue 1 https://doi.org/10.1111/gwat.12164	journal	February 2014
Probabilistic sensitivity analysis of complex models: a Bayesian approach Oakley, Jeremy E.; O'Hagan, Anthony Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3 https://doi.org/10.1111/j.1467-9868.2004.05304.x	journal	August 2004
Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches Borgonovo, Emanuele Risk Analysis, Vol. 26, Issue 5 https://doi.org/10.1111/j.1539-6924.2006.00806.x	journal	October 2006
Two-Dimensional Advective Transport in Ground-Water Flow Parameter Estimation Anderman, Evan R.; Hill, Mary C.; Poeter, Eileen P. Ground Water, Vol. 34, Issue 6 https://doi.org/10.1111/j.1745-6584.1996.tb02165.x	journal	November 1996
Are Models Too Simple? Arguments for Increased Parameterization Hunt, Randall J.; Doherty, John; Tonkin, Matthew J. Ground Water, Vol. 45, Issue 3 https://doi.org/10.1111/j.1745-6584.2007.00316.x	journal	May 2007
Testing Alternative Ground Water Models Using Cross-Validation and Other Methods Foglia, L.; Mehl, S. W.; Hill, M. C. Ground Water, Vol. 45, Issue 5 https://doi.org/10.1111/j.1745-6584.2007.00341.x	journal	September 2007
Quantifying Data Worth Toward Reducing Predictive Uncertainty Dausman, Alyssa M.; Doherty, John; Langevin, Christian D. Ground Water, Vol. 48, Issue 5 https://doi.org/10.1111/j.1745-6584.2010.00679.x	journal	August 2010
Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences Oreskes, N.; Shrader-Frechette, K.; Belitz, K. Science, Vol. 263, Issue 5147 https://doi.org/10.1126/science.263.5147.641	journal	February 1994
OPR-PPR, a computer program for assessing data importance to model predictions using linear statistics Tonkin, Matthew J.; Tiedeman, Claire; Ely, D. Matthew https://doi.org/10.3133/tm6E2	report	January 2007
Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation Tang, Y.; Reed, P.; Wagener, T. Hydrology and Earth System Sciences, Vol. 11, Issue 2 https://doi.org/10.5194/hess-11-793-2007	journal	January 2007
Flood forecasting using a fully distributed model: application of the TOPKAPI model to the Upper Xixian Catchment Liu, Z.; Martina, M. L. V.; Todini, E. Hydrology and Earth System Sciences, Vol. 9, Issue 4 https://doi.org/10.5194/hess-9-347-2005	journal	January 2005
Assessing parameter importance of the Common Land Model based on qualitative and quantitative sensitivity analysis Li, J. D.; Duan, Q. Y.; Gong, W. Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 2 https://doi.org/10.5194/hessd-10-2243-2013	journal	January 2013
Technical note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models Herman, J. D.; Kollat, J. B.; Reed, P. M. Hydrology and Earth System Sciences Discussions, Vol. 10, Issue 4 https://doi.org/10.5194/hessd-10-4275-2013	journal	January 2013

Cited By (15)

A Bayesian approach to improved calibration and prediction of groundwater models with structural error Xu, Tianfang; Valocchi, Albert J. Water Resources Research, Vol. 51, Issue 11 https://doi.org/10.1002/2015wr017912	journal	November 2015
Making the most out of a hydrological model data set: Sensitivity analyses to open the model black-box: ANALYSES TO OPEN THE MODEL BLACK-BOX Borgonovo, E.; Lu, X.; Plischke, E. Water Resources Research, Vol. 53, Issue 9 https://doi.org/10.1002/2017wr020767	journal	September 2017
Reconnecting Stochastic Methods With Hydrogeological Applications: A Utilitarian Uncertainty Analysis and Risk Assessment Approach for the Design of Optimal Monitoring Networks Bode, Felix; Ferré, Ty; Zigelli, Niklas Water Resources Research, Vol. 54, Issue 3 https://doi.org/10.1002/2017wr020919	journal	March 2018
Signature-Domain Calibration of Hydrological Models Using Approximate Bayesian Computation: Empirical Analysis of Fundamental Properties Fenicia, Fabrizio; Kavetski, Dmitri; Reichert, Peter Water Resources Research, Vol. 54, Issue 6 https://doi.org/10.1002/2017wr021616	journal	June 2018
Why Should Practitioners be Concerned about Predictive Uncertainty of Groundwater Management Models? Delottier, H.; Pryet, A.; Dupuy, A. Water Resources Management, Vol. 31, Issue 1 https://doi.org/10.1007/s11269-016-1508-2	journal	October 2016
The Fast and the Robust: Trade‐Offs Between Optimization Robustness and Cost in the Calibration of Environmental Models Kavetski, Dmitri; Qin, Youwei; Kuczera, George Water Resources Research, Vol. 54, Issue 11 https://doi.org/10.1029/2017wr022051	journal	November 2018
A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: From Standard Gauss‐Newton to Robust Gauss‐Newton Qin, Youwei; Kavetski, Dmitri; Kuczera, George Water Resources Research, Vol. 54, Issue 11 https://doi.org/10.1029/2017wr022488	journal	November 2018
A Robust Gauss‐Newton Algorithm for the Optimization of Hydrological Models: Benchmarking Against Industry‐Standard Algorithms Qin, Youwei; Kavetski, Dmitri; Kuczera, George Water Resources Research, Vol. 54, Issue 11 https://doi.org/10.1029/2017wr022489	journal	November 2018
Model Variable Augmentation (MVA) for Diagnostic Assessment of Sensitivity Analysis Results Mai, Juliane; Tolson, Bryan A. Water Resources Research, Vol. 55, Issue 4 https://doi.org/10.1029/2018wr023382	journal	April 2019
Sensitivity Analysis and Calibration of an Integrated Hydrologic Model in an Irrigated Agricultural Basin With a Groundwater‐Dependent Ecosystem Tolley, D.; Foglia, L.; Harter, T. Water Resources Research, Vol. 55, Issue 9 https://doi.org/10.1029/2018wr024209	journal	September 2019
Comparison of Newton-type and SCE optimisation algorithms for the calibration of conceptual hydrological models Qin, Youwei; Kuczera, George; Kavetski, Dmitri Australasian Journal of Water Resources, Vol. 20, Issue 2 https://doi.org/10.1080/13241583.2017.1298180	journal	July 2016
Terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution Barnhart, Katherine R.; Glade, Rachel C.; Shobe, Charles M. Geoscientific Model Development, Vol. 12, Issue 4 https://doi.org/10.5194/gmd-12-1267-2019	journal	January 2019
terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution Barnhart, Katherine R.; Glade, Rachel C.; Shobe, Charles M. Geoscientific Model Development https://doi.org/10.5194/gmd-2018-204	posted_content	October 2018
Characterizing and reducing equifinality by constraining a distributed catchment model with regional signatures, local observations, and process understanding Kelleher, Christa; McGlynn, Brian; Wagener, Thorsten Hydrology and Earth System Sciences, Vol. 21, Issue 7 https://doi.org/10.5194/hess-21-3325-2017	journal	January 2017
The evolution of process-based hydrologic models: historical challenges and the collective quest for physical realism Clark, Martyn P.; Bierkens, Marc F. P.; Samaniego, Luis Hydrology and Earth System Sciences, Vol. 21, Issue 7 https://doi.org/10.5194/hess-21-3427-2017	journal	January 2017

Similar Records

Resource frugal optimizer for quantum machine learning

Journal Article · Tue Aug 22 20:00:00 EDT 2023 · Quantum Science and Technology · OSTI ID:2228652

Evaluating marginal likelihood with thermodynamic integration method and comparison with several other numerical methods

Journal Article · Thu Feb 04 19:00:00 EST 2016 · Water Resources Research · OSTI ID:1326496

Related Subjects

54 ENVIRONMENTAL SCIENCES
97 MATHEMATICS AND COMPUTING
Bayesian
Model calibration
Sensitivity analysis
Uncertainty
falsifiability
regression
transparency

Practical Use of Computationally Frugal Model Analysis Methods

Citation Formats

References (49)

Cited By (15)

Similar Records

Related Subjects