Assessing the predictive impact of factor fixing with an adaptive uncertainty-based approach .

Wang, Qian; Guillaume, Joseph; Jakeman, John; Yang, Tao; Iwanaga, Takuya; Croke, Barry; Jakeman, Tony

doi:10.1016/j.envsoft.2021.105290

Title: Assessing the predictive impact of factor fixing with an adaptive uncertainty-based approach .

Journal Article · Thu Dec 16 00:00:00 EST 2021 · Environmental Modelling and Software

DOI:https://doi.org/10.1016/j.envsoft.2021.105290· OSTI ID:1845388

Wang, Qian ^[1]; Guillaume, Joseph ^[2]; Jakeman, John ^[3]; Yang, Tao ^[4]; Iwanaga, Takuya ^[2]; Croke, Barry ^[5]; Jakeman, Tony ^[2]

Hohai Univ., Nanjing (China). Center for Global Change and Water Cycle, State Key Lab. of Hydrology-Water Resources and Hydraulic Engineering; Australian National Univ., Canberra, ACT (Australia). Inst. for Water Futures
Australian National Univ., Canberra, ACT (Australia). Inst. for Water Futures
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Hohai Univ., Nanjing (China). Center for Global Change and Water Cycle, State Key Lab. of Hydrology-Water Resources and Hydraulic Engineering
Australian National Univ., Canberra, ACT (Australia). Inst. for Water Futures; Australian National Univ., Canberra, ACT (Australia). Mathematical Sciences Inst.

Despite widespread use of factor fixing in environmental modeling, its effect on model predictions has received little attention and is instead commonly presumed to be negligible. We propose a proof-of-concept adaptive method for systematically investigating the impact of factor fixing. The method uses Global Sensitivity Analysis methods to identify groups of sensitive parameters, then quantifies which groups can be safely fixed at nominal values without exceeding a maximum acceptable error, demonstrated using the 21-dimensional Sobol’ G-function. Furthermore, three error measures are considered for quantities of interest, namely Relative Mean Absolute Error, Pearson Product-Moment Correlation and Relative Variance. Results demonstrate that factor fixing may cause large errors in the model results unexpectedly, when preliminary analysis suggests otherwise, and that the default value selected affects the number of factors to fix. To improve the applicability and methodological development of factor fixing, a new research agenda encompassing five opportunities is discussed for further attention.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA); Australian Research Council (ARC); National Natural Science Foundation of China (NSFC); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: NA0003525; DE190100317; 51879068; NA-0003525

OSTI ID:: 1845388

Alternate ID(s):: OSTI ID: 1868839

Report Number(s):: SAND2022-1273J; 703223

Journal Information:: Environmental Modelling and Software, Vol. 148; ISSN 1364-8152

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (65)

Sensitivity analysis of the aquatic pesticide fate models in SYNOPS and their parametrization for Switzerland de Baan, Laura Science of The Total Environment, Vol. 715 https://doi.org/10.1016/j.scitotenv.2020.136881	journal	May 2020
A review of catchment-scale water quality and erosion models and a synthesis of future prospects Fu, Baihua; Merritt, Wendy S.; Croke, Barry F. W. Environmental Modelling & Software, Vol. 114 https://doi.org/10.1016/j.envsoft.2018.12.008	journal	April 2019
Model Evaluation Guidelines for Systematic Quantification of Accuracy in Watershed Simulations Transactions of the ASABE, Vol. 50, Issue 3 https://doi.org/10.13031/2013.23153	journal	January 2007
Bayesian framework of parameter sensitivity, uncertainty, and identifiability analysis in complex water quality models Jia, Haifeng; Xu, Te; Liang, Shidong Environmental Modelling & Software, Vol. 104 https://doi.org/10.1016/j.envsoft.2018.03.001	journal	June 2018
Sensitivity analysis approaches to high-dimensional screening problems at low sample size Becker, W. E.; Tarantola, S.; Deman, G. Journal of Statistical Computation and Simulation, Vol. 88, Issue 11 https://doi.org/10.1080/00949655.2018.1450876	journal	April 2018
Derivative-Based Global Sensitivity Analysis for Models with High-Dimensional Inputs and Functional Outputs Cleaves, Helen L.; Alexanderian, Alen; Guy, Hayley SIAM Journal on Scientific Computing, Vol. 41, Issue 6 https://doi.org/10.1137/19M1243518	journal	January 2019
Hydrologic modeling in dynamic catchments: A data assimilation approach Pathiraja, S.; Marshall, L.; Sharma, A. Water Resources Research, Vol. 52, Issue 5 https://doi.org/10.1002/2015WR017192	journal	May 2016
Recent insights on uncertainties present in integrated catchment water quality modelling Tscheikner-Gratl, Franz; Bellos, Vasilis; Schellart, Alma Water Research, Vol. 150 https://doi.org/10.1016/j.watres.2018.11.079	journal	March 2019
Factorial Sampling Plans for Preliminary Computational Experiments Morris, Max D. Technometrics, Vol. 33, Issue 2 https://doi.org/10.1080/00401706.1991.10484804	journal	May 1991
Time‐Varying Sensitivity Analysis Reveals Relationships Between Watershed Climate and Variations in Annual Parameter Importance in Regions With Strong Interannual Variability Basijokaite, R.; Kelleher, C. Water Resources Research, Vol. 57, Issue 1 https://doi.org/10.1029/2020WR028544	journal	January 2021
Climate change and probabilistic scenario of streamflow extremes in an alpine region: projection of streamflow extremes Yang, Tao; Wang, Xiaoyan; Yu, Zhongbo Journal of Geophysical Research: Atmospheres, Vol. 119, Issue 14 https://doi.org/10.1002/2014JD021824	journal	July 2014
Variance-based sensitivity analysis for time-dependent processes Alexanderian, Alen; Gremaud, Pierre A.; Smith, Ralph C. Reliability Engineering & System Safety, Vol. 196 https://doi.org/10.1016/j.ress.2019.106722	journal	April 2020
Computationally inexpensive identification of noninformative model parameters by sequential screening: INEXPENSIVE IDENTIFICATION OF NONINFORMATIVE PARAMETERS Cuntz, Matthias; Mai, Juliane; Zink, Matthias Water Resources Research, Vol. 51, Issue 8 https://doi.org/10.1002/2015WR016907	journal	August 2015
The stationarity paradigm revisited: Hypothesis testing using diagnostics, summary metrics, and DREAM _(ABC) Sadegh, Mojtaba; Vrugt, Jasper A.; Xu, Chonggang Water Resources Research, Vol. 51, Issue 11 https://doi.org/10.1002/2014WR016805	journal	November 2015
Tree-based ensemble methods for sensitivity analysis of environmental models: A performance comparison with Sobol and Morris techniques Jaxa-Rozen, Marc; Kwakkel, Jan Environmental Modelling & Software, Vol. 107 https://doi.org/10.1016/j.envsoft.2018.06.011	journal	September 2018
Estimating the approximation error when fixing unessential factors in global sensitivity analysis Sobol’, I. M.; Tarantola, S.; Gatelli, D. Reliability Engineering & System Safety, Vol. 92, Issue 7 https://doi.org/10.1016/j.ress.2006.07.001	journal	July 2007
A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application Razavi, Saman; Gupta, Hoshin V. Water Resources Research, Vol. 52, Issue 1 https://doi.org/10.1002/2015WR017559	journal	January 2016
A global sensitivity analysis approach for identifying critical sources of uncertainty in non-identifiable, spatially distributed environmental models: A holistic analysis applied to SWAT for input datasets and model parameters Koo, Hyeongmo; Chen, Min; Jakeman, Anthony J. Environmental Modelling & Software, Vol. 127 https://doi.org/10.1016/j.envsoft.2020.104676	journal	May 2020
Adaptive multi‐index collocation for uncertainty quantification and sensitivity analysis Jakeman, John D.; Eldred, Michael S.; Geraci, Gianluca International Journal for Numerical Methods in Engineering, Vol. 121, Issue 6 https://doi.org/10.1002/nme.6268	journal	November 2019
Three complementary methods for sensitivity analysis of a water quality model Sun, X. Y.; Newham, L. T. H.; Croke, B. F. W. Environmental Modelling & Software, Vol. 37 https://doi.org/10.1016/j.envsoft.2012.04.010	journal	November 2012
Technical Note: Method of Morris effectively reduces the computational demands of global sensitivity analysis for distributed watershed models Herman, J. D.; Kollat, J. B.; Reed, P. M. Hydrology and Earth System Sciences, Vol. 17, Issue 7 https://doi.org/10.5194/hess-17-2893-2013	journal	January 2013
Global sensitivity analysis in hydrological modeling: Review of concepts, methods, theoretical framework, and applications Song, Xiaomeng; Zhang, Jianyun; Zhan, Chesheng Journal of Hydrology, Vol. 523 https://doi.org/10.1016/j.jhydrol.2015.02.013	journal	April 2015
An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together? Maier, H. R.; Guillaume, J. H. A.; van Delden, H. Environmental Modelling & Software, Vol. 81 https://doi.org/10.1016/j.envsoft.2016.03.014	journal	July 2016
Global sensitivity analysis via multi-fidelity polynomial chaos expansion Palar, Pramudita Satria; Zuhal, Lavi Rizki; Shimoyama, Koji Reliability Engineering & System Safety, Vol. 170 https://doi.org/10.1016/j.ress.2017.10.013	journal	February 2018
Design of Experiment in Global Sensitivity Analysis Based on ANOVA High-Dimensional Model Representation Wang, Xiaodi; Tang, Yincai; Chen, Xueping Communications in Statistics - Simulation and Computation, Vol. 39, Issue 6 https://doi.org/10.1080/03610918.2010.484122	journal	June 2010
Evaluating uncertainty in integrated environmental models: A review of concepts and tools: EVALUATING ENVIRONMENTAL MODELS Matott, L. Shawn; Babendreier, Justin E.; Purucker, S. Thomas Water Resources Research, Vol. 45, Issue 6 https://doi.org/10.1029/2008WR007301	journal	June 2009
Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model Khorashadi Zadeh, Farkhondeh; Nossent, Jiri; Sarrazin, Fanny Environmental Modelling & Software, Vol. 91 https://doi.org/10.1016/j.envsoft.2017.02.001	journal	May 2017
Performance metrics for climate models Gleckler, P. J.; Taylor, K. E.; Doutriaux, C. Journal of Geophysical Research, Vol. 113, Issue D6 https://doi.org/10.1029/2007JD008972	journal	January 2008
Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates Sobol′, I. M. Mathematics and Computers in Simulation, Vol. 55, Issue 1-3 https://doi.org/10.1016/S0378-4754(00)00270-6	journal	February 2001
Sobol’ sensitivity analysis of a complex environmental model Nossent, Jiri; Elsen, Pieter; Bauwens, Willy Environmental Modelling & Software, Vol. 26, Issue 12 https://doi.org/10.1016/j.envsoft.2011.08.010	journal	December 2011
Addressing factors fixing setting from given data: A comparison of different methods Mara, Thierry A.; Belfort, Benjamin; Fontaine, Vincent Environmental Modelling & Software, Vol. 87 https://doi.org/10.1016/j.envsoft.2016.10.004	journal	January 2017
A method for evaluating the importance of system state observations to model predictions, with application to the Death Valley regional groundwater flow system: IMPORTANCE OF OBSERVATIONS TO MODEL PREDICTIONS Tiedeman, Claire R.; Ely, D. Matthew; Hill, Mary C. Water Resources Research, Vol. 40, Issue 12 https://doi.org/10.1029/2004WR003313	journal	December 2004
Characterization of an airflow network model by sensitivity analysis: parameter screening, fixing, prioritizing and mapping Monari, F.; Strachan, P. Journal of Building Performance Simulation, Vol. 10, Issue 1 https://doi.org/10.1080/19401493.2015.1110621	journal	January 2016
Analysis of variance designs for model output Jansen, Michiel J. W. Computer Physics Communications, Vol. 117, Issue 1-2 https://doi.org/10.1016/S0010-4655(98)00154-4	journal	March 1999
Global sensitivity analysis of an energy–economy model of the residential building sector Branger, Frédéric; Giraudet, Louis-Gaëtan; Guivarch, Céline Environmental Modelling & Software, Vol. 70 https://doi.org/10.1016/j.envsoft.2015.03.021	journal	August 2015
Global sensitivity analysis for high-dimensional problems: How to objectively group factors and measure robustness and convergence while reducing computational cost Sheikholeslami, Razi; Razavi, Saman; Gupta, Hoshin V. Environmental Modelling & Software, Vol. 111 https://doi.org/10.1016/j.envsoft.2018.09.002	journal	January 2019
An introduction to sensitivity assessment of simulation models Norton, John Environmental Modelling & Software, Vol. 69 https://doi.org/10.1016/j.envsoft.2015.03.020	journal	July 2015
A multi-criteria trajectory-based parameter sampling strategy for the screening method of elementary effects Khare, Y. P.; Muñoz-Carpena, R.; Rooney, R. W. Environmental Modelling & Software, Vol. 64 https://doi.org/10.1016/j.envsoft.2014.11.013	journal	February 2015
Global sensitivity analysis for urban water quality modelling: Terminology, convergence and comparison of different methods Vanrolleghem, Peter A.; Mannina, Giorgio; Cosenza, Alida Journal of Hydrology, Vol. 522 https://doi.org/10.1016/j.jhydrol.2014.12.056	journal	March 2015
Quasi-Monte Carlo based uncertainty analysis: Sampling efficiency and error estimation in engineering applications Hou, Tianfeng; Nuyens, Dirk; Roels, Staf Reliability Engineering & System Safety, Vol. 191 https://doi.org/10.1016/j.ress.2019.106549	journal	November 2019
Robust combination of the Morris and Sobol methods in complex multidimensional models Garcia, Dorleta; Arostegui, Inmaculada; Prellezo, Raúl Environmental Modelling & Software, Vol. 122 https://doi.org/10.1016/j.envsoft.2019.104517	journal	December 2019
Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose Guillaume, Joseph H. A.; Jakeman, John D.; Marsili-Libelli, Stefano Environmental Modelling & Software, Vol. 119 https://doi.org/10.1016/j.envsoft.2019.07.007	journal	September 2019
Global sensitivity analysis for medium-dimensional structural engineering problems using stochastic collocation Hübler, Clemens Reliability Engineering & System Safety, Vol. 195 https://doi.org/10.1016/j.ress.2019.106749	journal	March 2020
Parameter sensitivity analysis and optimization of Noah land surface model with field measurements from Huaihe River Basin, China Hou, Ting; Zhu, Yonghua; Lü, Haishen Stochastic Environmental Research and Risk Assessment, Vol. 29, Issue 5 https://doi.org/10.1007/s00477-015-1033-5	journal	February 2015
A review of surrogate models and their application to groundwater modeling: SURROGATES OF GROUNDWATER MODELS Asher, M. J.; Croke, B. F. W.; Jakeman, A. J. Water Resources Research, Vol. 51, Issue 8 https://doi.org/10.1002/2015WR016967	journal	August 2015
What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling Wagener, Thorsten; Pianosi, Francesca Earth-Science Reviews, Vol. 194 https://doi.org/10.1016/j.earscirev.2019.04.006	journal	July 2019
A structured analysis of uncertainty surrounding modeled impacts of groundwater-extraction rules Guillaume, Joseph H. A.; Qureshi, M. Ejaz; Jakeman, Anthony J. Hydrogeology Journal, Vol. 20, Issue 5 https://doi.org/10.1007/s10040-012-0864-0	journal	June 2012
Sensitivity of Whole-Body Dosimetry to Channel Model Parameters Jawad, Ourouk; Lautru, David; Benlarbi-Delai, Aziz IEEE Transactions on Antennas and Propagation, Vol. 63, Issue 8 https://doi.org/10.1109/TAP.2015.2434399	journal	August 2015
Global sensitivity analysis in wastewater applications: A comprehensive comparison of different methods Cosenza, Alida; Mannina, Giorgio; Vanrolleghem, Peter A. Environmental Modelling & Software, Vol. 49 https://doi.org/10.1016/j.envsoft.2013.07.009	journal	November 2013
Characterising performance of environmental models Bennett, Neil D.; Croke, Barry F. W.; Guariso, Giorgio Environmental Modelling & Software, Vol. 40 https://doi.org/10.1016/j.envsoft.2012.09.011	journal	February 2013
Making best use of model evaluations to compute sensitivity indices Saltelli, Andrea Computer Physics Communications, Vol. 145, Issue 2 https://doi.org/10.1016/S0010-4655(02)00280-1	journal	May 2002
Global Sensitivity Analysis of environmental models: Convergence and validation Sarrazin, Fanny; Pianosi, Francesca; Wagener, Thorsten Environmental Modelling & Software, Vol. 79 https://doi.org/10.1016/j.envsoft.2016.02.005	journal	May 2016
Sensitivity analysis of environmental models: A systematic review with practical workflow Pianosi, Francesca; Beven, Keith; Freer, Jim Environmental Modelling & Software, Vol. 79 https://doi.org/10.1016/j.envsoft.2016.02.008	journal	May 2016
Introductory overview: Error metrics for hydrologic modelling – A review of common practices and an open source library to facilitate use and adoption Jackson, Elise K.; Roberts, Wade; Nelsen, Benjamin Environmental Modelling & Software, Vol. 119 https://doi.org/10.1016/j.envsoft.2019.05.001	journal	September 2019
Emulation techniques for the reduction and sensitivity analysis of complex environmental models Ratto, Marco; Castelletti, Andrea; Pagano, Andrea Environmental Modelling & Software, Vol. 34 https://doi.org/10.1016/j.envsoft.2011.11.003	journal	June 2012
Variance based sensitivity analysis of model output. Design and estimator for the total sensitivity index Saltelli, Andrea; Annoni, Paola; Azzini, Ivano Computer Physics Communications, Vol. 181, Issue 2 https://doi.org/10.1016/j.cpc.2009.09.018	journal	February 2010
Methods for using groundwater model predictions to guide hydrogeologic data collection, with application to the Death Valley regional groundwater flow system: USING MODEL PREDICTIONS TO GUIDE DATA COLLECTION Tiedeman, Claire R.; Hill, Mary C.; D'Agnese, Frank A. Water Resources Research, Vol. 39, Issue 1 https://doi.org/10.1029/2001WR001255	journal	January 2003
Uncertainty analysis of a semi-distributed hydrologic model based on a Gaussian Process emulator Yang, Jing; Jakeman, Anthony; Fang, Gonghuan Environmental Modelling & Software, Vol. 101 https://doi.org/10.1016/j.envsoft.2017.11.037	journal	March 2018
Sensitivity-guided reduction of parametric dimensionality for multi-objective calibration of watershed models van Werkhoven, Kathryn; Wagener, Thorsten; Reed, Patrick Advances in Water Resources, Vol. 32, Issue 8 https://doi.org/10.1016/j.advwatres.2009.03.002	journal	August 2009
Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation Tang, Y.; Reed, P.; Wagener, T. Hydrology and Earth System Sciences, Vol. 11, Issue 2 https://doi.org/10.5194/hess-11-793-2007	journal	January 2007
Global sensitivity analysis measures the quality of parameter estimation: The case of soil parameters and a crop model Varella, Hubert; Guérif, Martine; Buis, Samuel Environmental Modelling & Software, Vol. 25, Issue 3 https://doi.org/10.1016/j.envsoft.2009.09.012	journal	March 2010
Parameter sensitivity analysis of crop growth models based on the extended Fourier Amplitude Sensitivity Test method Wang, Jing; Li, Xin; Lu, Ling Environmental Modelling & Software, Vol. 48 https://doi.org/10.1016/j.envsoft.2013.06.007	journal	October 2013
Sensitivity Analysis‐Based Automatic Parameter Calibration of the VIC Model for Streamflow Simulations Over China Gou, Jiaojiao; Miao, Chiyuan; Duan, Qingyun Water Resources Research, Vol. 56, Issue 1 https://doi.org/10.1029/2019WR025968	journal	January 2020
Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques Douglas-Smith, Dominique; Iwanaga, Takuya; Croke, Barry F. W. Environmental Modelling & Software, Vol. 124 https://doi.org/10.1016/j.envsoft.2019.104588	journal	February 2020
SALib: An open-source Python library for Sensitivity Analysis Herman, Jon; Usher, Will The Journal of Open Source Software, Vol. 2, Issue 9 https://doi.org/10.21105/joss.00097	journal	January 2017

Similar Records

Sensitivity analysis applied to SiC failure probability in TRISO modeled with BISON

Journal Article · Sat Oct 01 00:00:00 EDT 2022 · Progress in Nuclear Energy · OSTI ID:1845388

Ghezzi, Carlotta G.; Schappel, Daniel; Pastore, Giovanni; +2 more

A Decision-Relevant Factor-Fixing Framework: Application to Uncertainty Analysis of a High-Dimensional Water Quality Model

Journal Article · Mon Jul 31 00:00:00 EDT 2023 · Water Resources Research · OSTI ID:1845388

Wang, Qian; Guillaume, Joseph H. A.; Jakeman, John Davis; +5 more

A Comparative Review of Sensitivity and Uncertainty Analysis of Large-Scale Systems - II: Statistical Methods

Journal Article · Thu Jul 15 00:00:00 EDT 2004 · Nuclear Science and Engineering · OSTI ID:1845388

Cacuci, Dan G; Ionescu-Bujor, Mihaela

Related Subjects

54 ENVIRONMENTAL SCIENCES
sensitivity analysis
factor fixing
error measure
adaptive method
partial sort
ranking

Title: Assessing the predictive impact of factor fixing with an adaptive uncertainty-based approach .

Citation Formats

References (65)

Similar Records

Related Subjects