The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Razavi, Saman; Jakeman, Anthony; Saltelli, Andrea; Prieur, Clémentine; Iooss, Bertrand; Borgonovo, Emanuele; Plischke, Elmar; Lo Piano, Samuele; Iwanaga, Takuya; Becker, William; Tarantola, Stefano; Guillaume, Joseph H. A.; Jakeman, John; Gupta, Hoshin; Melillo, Nicola; Rabitti, Giovanni; Chabridon, Vincent; Duan, Qingyun; Sun, Xifu; Smith, Stefán; Sheikholeslami, Razi; Hosseini, Nasim; Asadzadeh, Masoud; Puy, Arnald; Kucherenko, Sergei; Maier, Holger R.

doi:10.1016/j.envsoft.2020.104954

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Journal Article · Mon Dec 14 23:00:00 EST 2020 · Environmental Modelling and Software

DOI:https://doi.org/10.1016/j.envsoft.2020.104954· OSTI ID:1760445

^[1]; Jakeman, Anthony ^[2]; Saltelli, Andrea ^[3]; Prieur, Clémentine ^[4]; ^[5]; Borgonovo, Emanuele ^[6]; Plischke, Elmar ^[7]; Lo Piano, Samuele ^[8]; Iwanaga, Takuya ^[2]; Becker, William ^[9]; ^[10]; Guillaume, Joseph H. A. ^[2]; ^[11]; Gupta, Hoshin ^[12]; ^[13]; Rabitti, Giovanni ^[14]; Chabridon, Vincent ^[5]; Duan, Qingyun ^[15]; ^[2]; Smith, Stefán ^[8] more »

Univ. of Saskatchewan, Saskatoon, SK (Canada)
Australian National Univ., Canberra, ACT (Australia)
Univ. Oberta de Catalunya (UOC), Barcelona (Spain)
Univ. of Grenoble Alpes (France)
Électricité de France (EDF), Chatou (France); SINCLAIR AI Lab., Saclay (France)
Bocconi Univ., Milan (Italy)
Technical Univ. of Clausthal, Clausthal-Zellerfeld (Germany)
Univ. of Reading, Reading (United Kingdom)
Freelance Consultant and Researcher, Ispra (Italy)
European Commission, Ispra (Italy). Joint Research Centre
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Univ. of Arizona, Tucson, AZ (United States)
Univ. degli Studi di Pavia (Italy)
Heriot-Watt Univ., Edinburgh, Scotland (United Kingdom)
Hohai Univ., Nanjing (China)
Univ. of Saskatchewan, Saskatoon, SK (Canada); Univ. of Oxford (United Kingdom)
Univ. of Manitoba, Winnipeg, MB (Canada)
Princeton Univ., NJ (United States); Bergen Univ. (Norway)
Imperial College, London (United Kingdom)
Univ. of Adelaide, SA (Australia)

Sensitivity analysis (SA) is en route to becoming an integral part of mathematical modeling. The tremendous potential benefits of SA are, however, yet to be fully realized, both for advancing mechanistic and data-driven modeling of human and natural systems, and in support of decision making. In this perspective paper, a multidisciplinary group of researchers and practitioners revisit the current status of SA, and outline research challenges in regard to both theoretical frameworks and their applications to solve real-world problems. Six areas are discussed that warrant further attention, including (1) structuring and standardizing SA as a discipline, (2) realizing the untapped potential of SA for systems modeling, (3) addressing the computational burden of SA, (4) progressing SA in the context of machine learning, (5) clarifying the relationship and role of SA to uncertainty quantification, and (6) evolving the use of SA in support of decision making. An outlook for the future of SA is provided that underlines how SA must underpin a wide variety of activities to better serve science and society.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)

Grant/Contract Number:: AC04-94AL85000; NA0003525

OSTI ID:: 1760445

Report Number(s):: SAND--2021-0040J; 693182

Journal Information:: Environmental Modelling and Software, Journal Name: Environmental Modelling and Software Vol. 137; ISSN 1364-8152

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (261)

A Bayesian approach to sensitivity analysis Felli, James C.; Hazen, Gordon B. Health Economics, Vol. 8, Issue 3 https://doi.org/10.1002/(SICI)1099-1050(199905)8:3<263::AID-HEC426>3.0.CO;2-S	journal	May 1999
Distributed Evaluation of Local Sensitivity Analysis (DELSA), with application to hydrologic models: Distributed evaluation of local sensitivity analysis Rakovec, O.; Hill, M. C.; Clark, M. P. Water Resources Research, Vol. 50, Issue 1 https://doi.org/10.1002/2013WR014063	journal	January 2014
Are we unnecessarily constraining the agility of complex process-based models? Mendoza, Pablo A.; Clark, Martyn P.; Barlage, Michael Water Resources Research, Vol. 51, Issue 1 https://doi.org/10.1002/2014WR015820	journal	January 2015
What do we mean by sensitivity analysis? The need for comprehensive characterization of “global” sensitivity in Earth and Environmental systems models: A Critical Look at Sensitivity Analysis Razavi, Saman; Gupta, Hoshin V. Water Resources Research, Vol. 51, Issue 5 https://doi.org/10.1002/2014WR016527	journal	May 2015
A new framework for comprehensive, robust, and efficient global sensitivity analysis: 1. Theory Razavi, Saman; Gupta, Hoshin V. Water Resources Research, Vol. 52, Issue 1 https://doi.org/10.1002/2015WR017558	journal	January 2016
A new framework for comprehensive, robust, and efficient global sensitivity analysis: 2. Application Razavi, Saman; Gupta, Hoshin V. Water Resources Research, Vol. 52, Issue 1 https://doi.org/10.1002/2015WR017559	journal	January 2016
Most computational hydrology is not reproducible, so is it really science?: REPRODUCIBLE COMPUTATIONAL HYDROLOGY Hutton, Christopher; Wagener, Thorsten; Freer, Jim Water Resources Research, Vol. 52, Issue 10 https://doi.org/10.1002/2016WR019285	journal	October 2016
Hyphenated hydrology: Interdisciplinary evolution of water resource science: EVOLUTION OF WATER RESOURCE SCIENCE McCurley, Kathryn L.; Jawitz, James W. Water Resources Research, Vol. 53, Issue 4 https://doi.org/10.1002/2016WR019835	journal	April 2017
Robustness Metrics: How Are They Calculated, When Should They Be Used and Why Do They Give Different Results? McPhail, C.; Maier, H. R.; Kwakkel, J. H. Earth's Future, Vol. 6, Issue 2 https://doi.org/10.1002/2017EF000649	journal	February 2018
Making the most out of a hydrological model data set: Sensitivity analyses to open the model black-box: ANALYSES TO OPEN THE MODEL BLACK-BOX Borgonovo, E.; Lu, X.; Plischke, E. Water Resources Research, Vol. 53, Issue 9 https://doi.org/10.1002/2017WR020767	journal	September 2017
Towards reduced uncertainty in conceptual rainfall-runoff modelling: dynamic identifiability analysis Wagener, T.; McIntyre, N.; Lees, M. J. Hydrological Processes, Vol. 17, Issue 2 https://doi.org/10.1002/hyp.1135	journal	January 2003
Multicriteria sensitivity analysis as a diagnostic tool for understanding model behaviour and characterizing model uncertainty Haghnegahdar, Amin; Razavi, Saman; Yassin, Fuad Hydrological Processes, Vol. 31, Issue 25 https://doi.org/10.1002/hyp.11358	journal	October 2017
Anthropocene flooding: Challenges for science and society Razavi, Saman; Gober, Patricia; Maier, Holger R. Hydrological Processes, Vol. 34, Issue 8 https://doi.org/10.1002/hyp.13723	journal	March 2020
Adaptive multi‐index collocation for uncertainty quantification and sensitivity analysis Jakeman, John D.; Eldred, Michael S.; Geraci, Gianluca International Journal for Numerical Methods in Engineering, Vol. 121, Issue 6 https://doi.org/10.1002/nme.6268	journal	November 2019
Sensitivity analysis and optimization of system dynamics models: Regression analysis and statistical design of experiments Kleijnen, Jack P. C. System Dynamics Review, Vol. 11, Issue 4 https://doi.org/10.1002/sdr.4260110403	journal	January 1995
Construction and Comparison of High-Dimensional Sobol' Generators Sobol', Ilya M.; Asotsky, Danil; Kreinin, Alexander Wilmott, Vol. 2011, Issue 56 https://doi.org/10.1002/wilm.10056	journal	November 2011
Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior: TIME-VARYING SENSITIVITY OF WATERSHED MODELS Herman, J. D.; Reed, P. M.; Wagener, T. Water Resources Research, Vol. 49, Issue 3 https://doi.org/10.1002/wrcr.20124	journal	March 2013
Measuring Statistical Dependence with Hilbert-Schmidt Norms Gretton, Arthur; Bousquet, Olivier; Smola, Alex Lecture Notes in Computer Science https://doi.org/10.1007/11564089_7	book	January 2005
The Design and Analysis of Computer Experiments Santner, Thomas J.; Williams, Brian J.; Notz, William I. Springer Series in Statistics https://doi.org/10.1007/978-1-4757-3799-8	book	January 2003
A review of techniques for parameter sensitivity analysis of environmental models Hamby, D. M. Environmental Monitoring and Assessment, Vol. 32, Issue 2 https://doi.org/10.1007/BF00547132	journal	September 1994
Use of some sensitivity criteria for choosing networks with good generalization ability Dimopoulos, Yannis; Bourret, Paul; Lek, Sovan Neural Processing Letters, Vol. 2, Issue 6 https://doi.org/10.1007/BF02309007	journal	December 1995
Can global sensitivity analysis steer the implementation of models for environmental assessments and decision-making? Tarantola, S.; Giglioli, N.; Jesinghaus, J. Stochastic Environmental Research and Risk Assessment (SERRA), Vol. 16, Issue 1 https://doi.org/10.1007/s00477-001-0085-x	journal	February 2002
Explaining prediction models and individual predictions with feature contributions Štrumbelj, Erik; Kononenko, Igor Knowledge and Information Systems, Vol. 41, Issue 3 https://doi.org/10.1007/s10115-013-0679-x	journal	August 2013
Gaussian process optimization with failures: classification and convergence proof Bachoc, François; Helbert, Céline; Picheny, Victor Journal of Global Optimization, Vol. 78, Issue 3 https://doi.org/10.1007/s10898-020-00920-0	journal	July 2020
First-Order Sensitivity Analysis for Hidden Neuron Selection in Layer-Wise Training of Networks Li, Bo; Chen, Cheng Neural Processing Letters, Vol. 48, Issue 2 https://doi.org/10.1007/s11063-017-9764-6	journal	December 2017
Identifying wildland fire ignition factors through sensitivity analysis of a neural network Vasilakos, Christos; Kalabokidis, Kostas; Hatzopoulos, John Natural Hazards, Vol. 50, Issue 1 https://doi.org/10.1007/s11069-008-9326-3	journal	December 2008
Design of computer experiments: space filling and beyond Pronzato, Luc; Müller, Werner G. Statistics and Computing, Vol. 22, Issue 3 https://doi.org/10.1007/s11222-011-9242-3	journal	April 2011
A new variable importance measure for random forests with missing data Hapfelmeier, Alexander; Hothorn, Torsten; Ulm, Kurt Statistics and Computing, Vol. 24, Issue 1 https://doi.org/10.1007/s11222-012-9349-1	journal	August 2012
Reliable error estimation for Sobol’ indices Jiménez Rugama, Lluís Antoni; Gilquin, Laurent Statistics and Computing, Vol. 28, Issue 4 https://doi.org/10.1007/s11222-017-9759-1	journal	June 2017
Nonparametric estimation of probabilistic sensitivity measures Antoniano-Villalobos, Isadora; Borgonovo, Emanuele; Lu, Xuefei Statistics and Computing, Vol. 30, Issue 2 https://doi.org/10.1007/s11222-019-09887-9	journal	August 2019
Ensembles vs. information theory: supporting science under uncertainty Nearing, Grey S.; Gupta, Hoshin V. Frontiers of Earth Science, Vol. 12, Issue 4 https://doi.org/10.1007/s11707-018-0709-9	journal	May 2018
Problematic Quantifications: a Critical Appraisal of Scenario Making for a Global ‘Sustainable’ Food Production Saltelli, Andrea; Piano, Samuele Lo Food Ethics, Vol. 1, Issue 2 https://doi.org/10.1007/s41055-017-0020-6	journal	August 2017
Science for the post-normal age Funtowicz, Silvio O.; Ravetz, Jerome R. Futures, Vol. 25, Issue 7 https://doi.org/10.1016/0016-3287(93)90022-L	journal	September 1993
Nonlinear sensitivity analysis of multiparameter model systems Cukier, R. I.; Levine, H. B.; Shuler, K. E. Journal of Computational Physics, Vol. 26, Issue 1 https://doi.org/10.1016/0021-9991(78)90097-9	journal	January 1978
On the distribution of points in a cube and the approximate evaluation of integrals Sobol', I. M. USSR Computational Mathematics and Mathematical Physics, Vol. 7, Issue 4 https://doi.org/10.1016/0041-5553(67)90144-9	journal	January 1967
Application of neural networks to modelling nonlinear relationships in ecology Lek, Sovan; Delacoste, Marc; Baran, Philippe Ecological Modelling, Vol. 90, Issue 1 https://doi.org/10.1016/0304-3800(95)00142-5	journal	September 1996
Multilayer feedforward networks are universal approximators Hornik, Kurt; Stinchcombe, Maxwell; White, Halbert Neural Networks, Vol. 2, Issue 5 https://doi.org/10.1016/0893-6080(89)90020-8	journal	January 1989
Importance measures in global sensitivity analysis of nonlinear models Homma, Toshimitsu; Saltelli, Andrea Reliability Engineering & System Safety, Vol. 52, Issue 1 https://doi.org/10.1016/0951-8320(96)00002-6	journal	April 1996
Making best use of model evaluations to compute sensitivity indices Saltelli, Andrea Computer Physics Communications, Vol. 145, Issue 2 https://doi.org/10.1016/S0010-4655(02)00280-1	journal	May 2002
Avoidable waste in the production and reporting of research evidence Chalmers, Iain; Glasziou, Paul The Lancet, Vol. 374, Issue 9683 https://doi.org/10.1016/S0140-6736(09)60329-9	journal	July 2009
Neural network models to study relationships between lead concentration in grasses and permanent urban descriptors in Athens city (Greece) Dimopoulos, Ioannis; Chronopoulos, J.; Chronopoulou-Sereli, A. Ecological Modelling, Vol. 120, Issue 2-3 https://doi.org/10.1016/S0304-3800(99)00099-X	journal	August 1999
A novel feature selection method based on global sensitivity analysis with application in machine learning-based prediction model Zhang, Pin Applied Soft Computing, Vol. 85 https://doi.org/10.1016/j.asoc.2019.105859	journal	December 2019
Polynomial chaos expansions for dependent random variables Jakeman, John D.; Franzelin, Fabian; Narayan, Akil Computer Methods in Applied Mechanics and Engineering, Vol. 351 https://doi.org/10.1016/j.cma.2019.03.049	journal	July 2019
From screening to quantitative sensitivity analysis. A unified approach Campolongo, Francesca; Saltelli, Andrea; Cariboni, Jessica Computer Physics Communications, Vol. 182, Issue 4 https://doi.org/10.1016/j.cpc.2010.12.039	journal	April 2011
Estimation of global sensitivity indices for models with dependent variables Kucherenko, S.; Tarantola, S.; Annoni, P. Computer Physics Communications, Vol. 183, Issue 4 https://doi.org/10.1016/j.cpc.2011.12.020	journal	April 2012
Metamodelling with independent and dependent inputs Zuniga, M. Munoz; Kucherenko, S.; Shah, N. Computer Physics Communications, Vol. 184, Issue 6 https://doi.org/10.1016/j.cpc.2013.02.005	journal	June 2013
Iterative construction of replicated designs based on Sobol' sequences Gilquin, Laurent; Jiménez Rugama, Lluís Antoni; Arnaud, Élise Comptes Rendus Mathematique, Vol. 355, Issue 1 https://doi.org/10.1016/j.crma.2016.11.013	journal	January 2017
What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling Wagener, Thorsten; Pianosi, Francesca Earth-Science Reviews, Vol. 194 https://doi.org/10.1016/j.earscirev.2019.04.006	journal	July 2019
Questioning the Ecological Footprint Galli, Alessandro; Giampietro, Mario; Goldfinger, Steve Ecological Indicators, Vol. 69 https://doi.org/10.1016/j.ecolind.2016.04.014	journal	October 2016
Predicting diatom reference communities at the French hydrosystem scale: A first step towards the definition of the good ecological status Tison, J.; Park, Y. -S.; Coste, M. Ecological Modelling, Vol. 203, Issue 1-2 https://doi.org/10.1016/j.ecolmodel.2006.02.047	journal	April 2007
Scenario analysis in environmental impact assessment: Improving explorations of the future Duinker, Peter N.; Greig, Lorne A. Environmental Impact Assessment Review, Vol. 27, Issue 3 https://doi.org/10.1016/j.eiar.2006.11.001	journal	April 2007
Global sensitivity measures from given data Plischke, Elmar; Borgonovo, Emanuele; Smith, Curtis L. European Journal of Operational Research, Vol. 226, Issue 3 https://doi.org/10.1016/j.ejor.2012.11.047	journal	May 2013
Sensitivity analysis: A review of recent advances Borgonovo, Emanuele; Plischke, Elmar European Journal of Operational Research, Vol. 248, Issue 3 https://doi.org/10.1016/j.ejor.2015.06.032	journal	February 2016
Probabilistic sensitivity measures as information value Borgonovo, Emanuele; Hazen, Gordon B.; Jose, Victor Richmond R. European Journal of Operational Research, Vol. 289, Issue 2 https://doi.org/10.1016/j.ejor.2020.07.010	journal	March 2021
Parameter identification and sensitivity analysis of solar cell models with cat swarm optimization algorithm Guo, Lei; Meng, Zhuo; Sun, Yize Energy Conversion and Management, Vol. 108 https://doi.org/10.1016/j.enconman.2015.11.041	journal	January 2016
Decision making in contexts of deep uncertainty - An alternative approach for long-term climate policy Workman, Mark; Dooley, Kate; Lomax, Guy Environmental Science & Policy, Vol. 103 https://doi.org/10.1016/j.envsci.2019.10.002	journal	January 2020
Distance-based and stochastic uncertainty analysis for multi-criteria decision analysis in Excel using Visual Basic for Applications Hyde, K. M.; Maier, H. R. Environmental Modelling & Software, Vol. 21, Issue 12 https://doi.org/10.1016/j.envsoft.2005.08.004	journal	December 2006
Ten iterative steps in development and evaluation of environmental models Jakeman, A. J.; Letcher, R. A.; Norton, J. P. Environmental Modelling & Software, Vol. 21, Issue 5 https://doi.org/10.1016/j.envsoft.2006.01.004	journal	May 2006
A formal framework for scenario development in support of environmental decision-making Mahmoud, Mohammed; Liu, Yuqiong; Hartmann, Holly Environmental Modelling & Software, Vol. 24, Issue 7 https://doi.org/10.1016/j.envsoft.2008.11.010	journal	July 2009
Management Option Rank Equivalence (MORE) – A new method of sensitivity analysis for decision-making Ravalico, J. K.; Dandy, G. C.; Maier, H. R. Environmental Modelling & Software, Vol. 25, Issue 2 https://doi.org/10.1016/j.envsoft.2009.06.012	journal	February 2010
Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions Maier, Holger R.; Jain, Ashu; Dandy, Graeme C. Environmental Modelling & Software, Vol. 25, Issue 8 https://doi.org/10.1016/j.envsoft.2010.02.003	journal	August 2010
How to avoid a perfunctory sensitivity analysis Saltelli, Andrea; Annoni, Paola Environmental Modelling & Software, Vol. 25, Issue 12 https://doi.org/10.1016/j.envsoft.2010.04.012	journal	December 2010
A comprehensive evaluation of various sensitivity analysis methods: A case study with a hydrological model Gan, Yanjun; Duan, Qingyun; Gong, Wei Environmental Modelling & Software, Vol. 51 https://doi.org/10.1016/j.envsoft.2013.09.031	journal	January 2014
Protocol for developing ANN models and its application to the assessment of the quality of the ANN model development process in drinking water quality modelling Wu, Wenyan; Dandy, Graeme C.; Maier, Holger R. Environmental Modelling & Software, Vol. 54 https://doi.org/10.1016/j.envsoft.2013.12.016	journal	April 2014
An evaluation framework for input variable selection algorithms for environmental data-driven models Galelli, Stefano; Humphrey, Greer B.; Maier, Holger R. Environmental Modelling & Software, Vol. 62 https://doi.org/10.1016/j.envsoft.2014.08.015	journal	December 2014
Evolutionary algorithms and other metaheuristics in water resources: Current status, research challenges and future directions Maier, H. R.; Kapelan, Z.; Kasprzyk, J. Environmental Modelling & Software, Vol. 62 https://doi.org/10.1016/j.envsoft.2014.09.013	journal	December 2014
A simple and efficient method for global sensitivity analysis based on cumulative distribution functions Pianosi, Francesca; Wagener, Thorsten Environmental Modelling & Software, Vol. 67 https://doi.org/10.1016/j.envsoft.2015.01.004	journal	May 2015
An introduction to sensitivity assessment of simulation models Norton, John Environmental Modelling & Software, Vol. 69 https://doi.org/10.1016/j.envsoft.2015.03.020	journal	July 2015
A Matlab toolbox for Global Sensitivity Analysis Pianosi, Francesca; Sarrazin, Fanny; Wagener, Thorsten Environmental Modelling & Software, Vol. 70 https://doi.org/10.1016/j.envsoft.2015.04.009	journal	August 2015
A modified Sobol′ sensitivity analysis method for decision-making in environmental problems Ganji, Arman; Maier, Holger R.; Dandy, Graeme C. Environmental Modelling & Software, Vol. 75 https://doi.org/10.1016/j.envsoft.2015.10.001	journal	January 2016
Global Sensitivity Analysis of environmental models: Convergence and validation Sarrazin, Fanny; Pianosi, Francesca; Wagener, Thorsten Environmental Modelling & Software, Vol. 79 https://doi.org/10.1016/j.envsoft.2016.02.005	journal	May 2016
Sensitivity analysis of environmental models: A systematic review with practical workflow Pianosi, Francesca; Beven, Keith; Freer, Jim Environmental Modelling & Software, Vol. 79 https://doi.org/10.1016/j.envsoft.2016.02.008	journal	May 2016
An uncertain future, deep uncertainty, scenarios, robustness and adaptation: How do they fit together? Maier, H. R.; Guillaume, J. H. A.; van Delden, H. Environmental Modelling & Software, Vol. 81 https://doi.org/10.1016/j.envsoft.2016.03.014	journal	July 2016
Robust discrimination between uncertain management alternatives by iterative reflection on crossover point scenarios: Principles, design and implementations Guillaume, Joseph H. A.; Arshad, Muhammad; Jakeman, Anthony J. Environmental Modelling & Software, Vol. 83 https://doi.org/10.1016/j.envsoft.2016.04.005	journal	September 2016
Improved validation framework and R-package for artificial neural network models Humphrey, Greer B.; Maier, Holger R.; Wu, Wenyan Environmental Modelling & Software, Vol. 92 https://doi.org/10.1016/j.envsoft.2017.01.023	journal	June 2017
Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model Khorashadi Zadeh, Farkhondeh; Nossent, Jiri; Sarrazin, Fanny Environmental Modelling & Software, Vol. 91 https://doi.org/10.1016/j.envsoft.2017.02.001	journal	May 2017
Progressive Latin Hypercube Sampling: An efficient approach for robust sampling-based analysis of environmental models Sheikholeslami, Razi; Razavi, Saman Environmental Modelling & Software, Vol. 93 https://doi.org/10.1016/j.envsoft.2017.03.010	journal	July 2017
Robust optimization of water infrastructure planning under deep uncertainty using metamodels Beh, Eva H. Y.; Zheng, Feifei; Dandy, Graeme C. Environmental Modelling & Software, Vol. 93 https://doi.org/10.1016/j.envsoft.2017.03.013	journal	July 2017
Insights into sensitivity analysis of Earth and environmental systems models: On the impact of parameter perturbation scale Haghnegahdar, Amin; Razavi, Saman Environmental Modelling & Software, Vol. 95 https://doi.org/10.1016/j.envsoft.2017.03.031	journal	September 2017
Uncertainty analysis of a semi-distributed hydrologic model based on a Gaussian Process emulator Yang, Jing; Jakeman, Anthony; Fang, Gonghuan Environmental Modelling & Software, Vol. 101 https://doi.org/10.1016/j.envsoft.2017.11.037	journal	March 2018
Distribution-based sensitivity analysis from a generic input-output sample Pianosi, Francesca; Wagener, Thorsten Environmental Modelling & Software, Vol. 108 https://doi.org/10.1016/j.envsoft.2018.07.019	journal	October 2018
Global sensitivity analysis for high-dimensional problems: How to objectively group factors and measure robustness and convergence while reducing computational cost Sheikholeslami, Razi; Razavi, Saman; Gupta, Hoshin V. Environmental Modelling & Software, Vol. 111 https://doi.org/10.1016/j.envsoft.2018.09.002	journal	January 2019
VARS-TOOL: A toolbox for comprehensive, efficient, and robust sensitivity and uncertainty analysis Razavi, Saman; Sheikholeslami, Razi; Gupta, Hoshin V. Environmental Modelling & Software, Vol. 112 https://doi.org/10.1016/j.envsoft.2018.10.005	journal	February 2019
Introductory overview: Optimization using evolutionary algorithms and other metaheuristics Maier, H. R.; Razavi, S.; Kapelan, Z. Environmental Modelling & Software, Vol. 114 https://doi.org/10.1016/j.envsoft.2018.11.018	journal	April 2019
A multi-method Generalized Global Sensitivity Matrix approach to accounting for the dynamical nature of earth and environmental systems models Razavi, Saman; Gupta, Hoshin V. Environmental Modelling & Software, Vol. 114 https://doi.org/10.1016/j.envsoft.2018.12.002	journal	April 2019
Why so many published sensitivity analyses are false: A systematic review of sensitivity analysis practices Saltelli, Andrea; Aleksankina, Ksenia; Becker, William Environmental Modelling & Software, Vol. 114 https://doi.org/10.1016/j.envsoft.2019.01.012	journal	April 2019
Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose Guillaume, Joseph H. A.; Jakeman, John D.; Marsili-Libelli, Stefano Environmental Modelling & Software, Vol. 119 https://doi.org/10.1016/j.envsoft.2019.07.007	journal	September 2019
Benchmarking the PAWN distribution-based method against the variance-based method in global sensitivity analysis: Empirical results Mora, Esteve Borràs; Spelling, James; van der Weijde, Adriaan H. Environmental Modelling & Software, Vol. 122 https://doi.org/10.1016/j.envsoft.2019.104556	journal	December 2019
How successfully is open-source research software adopted? Results and implications of surveying the users of a sensitivity analysis toolbox Pianosi, Francesca; Sarrazin, Fanny; Wagener, Thorsten Environmental Modelling & Software, Vol. 124 https://doi.org/10.1016/j.envsoft.2019.104579	journal	February 2020
Certain trends in uncertainty and sensitivity analysis: An overview of software tools and techniques Douglas-Smith, Dominique; Iwanaga, Takuya; Croke, Barry F. W. Environmental Modelling & Software, Vol. 124 https://doi.org/10.1016/j.envsoft.2019.104588	journal	February 2020
Evaluation of parameter interaction effect of hydrological models using the sparse polynomial chaos (SPC) method Wang, Heng; Gong, Wei; Duan, Qingyun Environmental Modelling & Software, Vol. 125 https://doi.org/10.1016/j.envsoft.2019.104612	journal	March 2020
A global sensitivity analysis approach for identifying critical sources of uncertainty in non-identifiable, spatially distributed environmental models: A holistic analysis applied to SWAT for input datasets and model parameters Koo, Hyeongmo; Chen, Min; Jakeman, Anthony J. Environmental Modelling & Software, Vol. 127 https://doi.org/10.1016/j.envsoft.2020.104676	journal	May 2020
A sensitivity analysis of the PAWN sensitivity index Puy, Arnald; Lo Piano, Samuele; Saltelli, Andrea Environmental Modelling & Software, Vol. 127 https://doi.org/10.1016/j.envsoft.2020.104679	journal	May 2020
Position paper: Sensitivity analysis of spatially distributed environmental models- a pragmatic framework for the exploration of uncertainty sources Koo, Hyeongmo; Iwanaga, Takuya; Croke, Barry F. W. Environmental Modelling & Software, Vol. 134 https://doi.org/10.1016/j.envsoft.2020.104857	journal	December 2020
Socio-technical scales in socio-environmental modeling: Managing a system-of-systems modeling approach Iwanaga, Takuya; Wang, Hsiao-Hsuan; Hamilton, Serena H. Environmental Modelling & Software, Vol. 135 https://doi.org/10.1016/j.envsoft.2020.104885	journal	January 2021
Enhancing the policy relevance of exploratory scenarios: Generic approach and application to disaster risk reduction Riddell, Graeme A.; van Delden, Hedwig; Dandy, Graeme C. Futures, Vol. 99 https://doi.org/10.1016/j.futures.2018.03.006	journal	May 2018
Why science’s crisis should not become a political battling ground Andrea, Saltelli Futures, Vol. 104 https://doi.org/10.1016/j.futures.2018.07.006	journal	December 2018
Nutrition and public health economic evaluations under the lenses of post normal science Lo Piano, Samuele; Robinson, Marguerite Futures, Vol. 112 https://doi.org/10.1016/j.futures.2019.06.008	journal	September 2019
Exploratory scenario analysis for disaster risk reduction: Considering alternative pathways in disaster risk assessment Riddell, Graeme A.; van Delden, Hedwig; Maier, Holger R. International Journal of Disaster Risk Reduction, Vol. 39 https://doi.org/10.1016/j.ijdrr.2019.101230	journal	October 2019
Bayesian sensitivity analysis of a model of the aortic valve Becker, W.; Rowson, J.; Oakley, J. E. Journal of Biomechanics, Vol. 44, Issue 8 https://doi.org/10.1016/j.jbiomech.2011.03.008	journal	May 2011
Open Science now: A systematic literature review for an integrated definition Vicente-Saez, Ruben; Martinez-Fuentes, Clara Journal of Business Research, Vol. 88 https://doi.org/10.1016/j.jbusres.2017.12.043	journal	July 2018
Nonnegative tensor factorization for contaminant source identification Vesselinov, Velimir V.; Alexandrov, Boian S.; O'Malley, Daniel Journal of Contaminant Hydrology, Vol. 220 https://doi.org/10.1016/j.jconhyd.2018.11.010	journal	January 2019
Optimal experimental design for prediction based on push-forward probability measures Butler, T.; Jakeman, J. D.; Wildey, T. Journal of Computational Physics, Vol. 416 https://doi.org/10.1016/j.jcp.2020.109518	journal	September 2020
Calibration and validation of neural networks to ensure physically plausible hydrological modeling Kingston, Greer B.; Maier, Holger R.; Lambert, Martin F. Journal of Hydrology, Vol. 314, Issue 1-4 https://doi.org/10.1016/j.jhydrol.2005.03.013	journal	November 2005
Addressing ten questions about conceptual rainfall–runoff models with global sensitivity analyses in R Shin, Mun-Ju; Guillaume, Joseph H. A.; Croke, Barry F. W. Journal of Hydrology, Vol. 503 https://doi.org/10.1016/j.jhydrol.2013.08.047	journal	October 2013
A review of foundational methods for checking the structural identifiability of models: Results for rainfall-runoff Shin, Mun-Ju; Guillaume, Joseph H. A.; Croke, Barry F. W. Journal of Hydrology, Vol. 520 https://doi.org/10.1016/j.jhydrol.2014.11.040	journal	January 2015
An inverse approach to perturb historical rainfall data for scenario-neutral climate impact studies Guo, Danlu; Westra, Seth; Maier, Holger R. Journal of Hydrology, Vol. 556 https://doi.org/10.1016/j.jhydrol.2016.03.025	journal	January 2018
Generating realistic perturbed hydrometeorological time series to inform scenario-neutral climate impact assessments Culley, S.; Bennett, B.; Westra, S. Journal of Hydrology, Vol. 576 https://doi.org/10.1016/j.jhydrol.2019.06.005	journal	September 2019
Worth of hydraulic and water chemistry observation data in terms of the reliability of surface water-groundwater exchange flux predictions under varied flow conditions Partington, Daniel; Knowling, Matthew J.; Simmons, Craig T. Journal of Hydrology, Vol. 590 https://doi.org/10.1016/j.jhydrol.2020.125441	journal	November 2020
Derivative based global sensitivity measures and their link with global sensitivity indices Sobol’, I. M.; Kucherenko, S. Mathematics and Computers in Simulation, Vol. 79, Issue 10 https://doi.org/10.1016/j.matcom.2009.01.023	journal	June 2009
Derivative-based global sensitivity measures: General links with Sobol’ indices and numerical tests Lamboni, M.; Iooss, B.; Popelin, A. -L. Mathematics and Computers in Simulation, Vol. 87 https://doi.org/10.1016/j.matcom.2013.02.002	journal	January 2013
A global sensitivity analysis approach for marine biogeochemical modeling Prieur, C.; Viry, L.; Blayo, E. Ocean Modelling, Vol. 139 https://doi.org/10.1016/j.ocemod.2019.101402	journal	July 2019
Co-designing Indus Water-Energy-Land Futures Wada, Yoshihide; Vinca, Adriano; Parkinson, Simon One Earth, Vol. 1, Issue 2 https://doi.org/10.1016/j.oneear.2019.10.006	journal	October 2019
A new uncertainty importance measure Borgonovo, E. Reliability Engineering & System Safety, Vol. 92, Issue 6 https://doi.org/10.1016/j.ress.2006.04.015	journal	June 2007
Estimating the approximation error when fixing unessential factors in global sensitivity analysis Sobol’, I. M.; Tarantola, S.; Gatelli, D. Reliability Engineering & System Safety, Vol. 92, Issue 7 https://doi.org/10.1016/j.ress.2006.07.001	journal	July 2007
Global sensitivity analysis using polynomial chaos expansions Sudret, Bruno Reliability Engineering & System Safety, Vol. 93, Issue 7 https://doi.org/10.1016/j.ress.2007.04.002	journal	July 2008
Uncertainty and sensitivity analysis for models with correlated parameters Xu, Chonggang; Gertner, George Zdzislaw Reliability Engineering & System Safety, Vol. 93, Issue 10 https://doi.org/10.1016/j.ress.2007.06.003	journal	October 2008
Calculations of Sobol indices for the Gaussian process metamodel Marrel, Amandine; Iooss, Bertrand; Laurent, Béatrice Reliability Engineering & System Safety, Vol. 94, Issue 3 https://doi.org/10.1016/j.ress.2008.07.008	journal	March 2009
Sensitivity analysis for decision-making using the MORE method—A Pareto approach Ravalico, Jakin K.; Maier, Holger R.; Dandy, Graeme C. Reliability Engineering & System Safety, Vol. 94, Issue 7 https://doi.org/10.1016/j.ress.2009.01.009	journal	July 2009
An effective algorithm for computing global sensitivity indices (EASI) Plischke, Elmar Reliability Engineering & System Safety, Vol. 95, Issue 4 https://doi.org/10.1016/j.ress.2009.11.005	journal	April 2010
The identification of model effective dimensions using global sensitivity analysis Kucherenko, Sergei; Feil, Balazs; Shah, Nilay Reliability Engineering & System Safety, Vol. 96, Issue 4 https://doi.org/10.1016/j.ress.2010.11.003	journal	April 2011
Global sensitivity analysis using sparse grid interpolation and polynomial chaos Buzzard, Gregery T. Reliability Engineering & System Safety, Vol. 107 https://doi.org/10.1016/j.ress.2011.07.011	journal	November 2012
Variable importance analysis: A comprehensive review Wei, Pengfei; Lu, Zhenzhou; Song, Jingwen Reliability Engineering & System Safety, Vol. 142 https://doi.org/10.1016/j.ress.2015.05.018	journal	October 2015
Sobol' indices for problems defined in non-rectangular domains Kucherenko, S.; Klymenko, O. V.; Shah, N. Reliability Engineering & System Safety, Vol. 167 https://doi.org/10.1016/j.ress.2017.06.001	journal	November 2017
Global sensitivity analysis via multi-fidelity polynomial chaos expansion Palar, Pramudita Satria; Zuhal, Lavi Rizki; Shimoyama, Koji Reliability Engineering & System Safety, Vol. 170 https://doi.org/10.1016/j.ress.2017.10.013	journal	February 2018
Reliability-based sensitivity estimators of rare event probability in the presence of distribution parameter uncertainty Chabridon, Vincent; Balesdent, Mathieu; Bourinet, Jean-Marc Reliability Engineering & System Safety, Vol. 178 https://doi.org/10.1016/j.ress.2018.06.008	journal	October 2018
Global sensitivity analysis in the context of imprecise probabilities (p-boxes) using sparse polynomial chaos expansions Schöbi, Roland; Sudret, Bruno Reliability Engineering & System Safety, Vol. 187 https://doi.org/10.1016/j.ress.2018.11.021	journal	July 2019
Metafunctions for benchmarking in sensitivity analysis Becker, William Reliability Engineering & System Safety, Vol. 204 https://doi.org/10.1016/j.ress.2020.107189	journal	December 2020
Nonparametric estimation of aggregated Sobol’ indices: Application to a depth averaged snow avalanche model Heredia, María Belén; Prieur, Clémentine; Eckert, Nicolas Reliability Engineering & System Safety, Vol. 212 https://doi.org/10.1016/j.ress.2020.107422	journal	August 2021
Trends in sensitivity analysis practice in the last decade Ferretti, Federico; Saltelli, Andrea; Tarantola, Stefano Science of The Total Environment, Vol. 568 https://doi.org/10.1016/j.scitotenv.2016.02.133	journal	October 2016
Scenario processes for socio-environmental systems analysis of futures: A review of recent efforts and a salient research agenda for supporting decision making Elsawah, Sondoss; Hamilton, Serena H.; Jakeman, Anthony J. Science of The Total Environment, Vol. 729 https://doi.org/10.1016/j.scitotenv.2020.138393	journal	August 2020
Influence of input PDF parameters of a model on a failure probability estimation Morio, Jérôme Simulation Modelling Practice and Theory, Vol. 19, Issue 10 https://doi.org/10.1016/j.simpat.2011.08.003	journal	November 2011
The future shocks: On discontinuity and scenario development van Notten, Ph. W. F.; Sleegers, A. M.; van Asselt, M. B. A. Technological Forecasting and Social Change, Vol. 72, Issue 2 https://doi.org/10.1016/j.techfore.2003.12.003	journal	February 2005
Kernel techniques: From machine learning to meshless methods Schaback, Robert; Wendland, Holger Acta Numerica, Vol. 15 https://doi.org/10.1017/S0962492906270016	journal	May 2006
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
A Taxonomy of Global Optimization Methods Based on Response Surfaces Jones, Donald R. Journal of Global Optimization, Vol. 21, Issue 4, p. 345-383 https://doi.org/10.1023/A:1012771025575	journal	December 2001
A hybrid regularized inversion methodology for highly parameterized environmental models: HYBRID REGULARIZATION METHODOLOGY Tonkin, Matthew James; Doherty, John Water Resources Research, Vol. 41, Issue 10 https://doi.org/10.1029/2005WR003995	journal	October 2005
Bayesian training of artificial neural networks used for water resources modeling: BAYESIAN TRAINING OF ANNS IN WATER RESOURCES MODELING Kingston, Greer B.; Lambert, Martin F.; Maier, Holger R. Water Resources Research, Vol. 41, Issue 12 https://doi.org/10.1029/2005WR004152	journal	December 2005
Bayesian model selection applied to artificial neural networks used for water resources modeling: BMS OF ANNS IN WATER RESOURCES MODELING Kingston, Greer B.; Maier, Holger R.; Lambert, Martin F. Water Resources Research, Vol. 44, Issue 4 https://doi.org/10.1029/2007WR006155	journal	April 2008
Calibration-constrained Monte Carlo analysis of highly parameterized models using subspace techniques: CALIBRATION-CONSTRAINED SSMC ANALYSIS Tonkin, Matthew; Doherty, John Water Resources Research, Vol. 45, Issue 12 https://doi.org/10.1029/2007WR006678	journal	January 2009
Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction Kavetski, Dmitri; Clark, Martyn P. Water Resources Research, Vol. 46, Issue 10 https://doi.org/10.1029/2009WR008896	journal	October 2010
Review of surrogate modeling in water resources: REVIEW Razavi, Saman; Tolson, Bryan A.; Burn, Donald H. Water Resources Research, Vol. 48, Issue 7 https://doi.org/10.1029/2011WR011527	journal	July 2012
Revisiting the Basis of Sensitivity Analysis for Dynamical Earth System Models Gupta, Hoshin V.; Razavi, Saman Water Resources Research, Vol. 54, Issue 11 https://doi.org/10.1029/2018WR022668	journal	November 2018
Legacy, Rather Than Adequacy, Drives the Selection of Hydrological Models Addor, N.; Melsen, L. A. Water Resources Research, Vol. 55, Issue 1 https://doi.org/10.1029/2018WR022958	journal	January 2019
Model Variable Augmentation (MVA) for Diagnostic Assessment of Sensitivity Analysis Results Mai, Juliane; Tolson, Bryan A. Water Resources Research, Vol. 55, Issue 4 https://doi.org/10.1029/2018WR023382	journal	April 2019
Correlation Effects? A Major but Often Neglected Component in Sensitivity and Uncertainty Analysis Do, Nhu Cuong; Razavi, Saman Water Resources Research, Vol. 56, Issue 3 https://doi.org/10.1029/2019WR025436	journal	March 2020
Impact of Scenario Selection on Robustness McPhail, C.; Maier, H. R.; Westra, S. Water Resources Research, Vol. 56, Issue 9 https://doi.org/10.1029/2019WR026515	journal	August 2020
Current Models Underestimate Future Irrigated Areas Puy, A.; Lo Piano, S.; Saltelli, A. Geophysical Research Letters, Vol. 47, Issue 8 https://doi.org/10.1029/2020GL087360	journal	April 2020
A Fresh Look at Variography: Measuring Dependence and Possible Sensitivities Across Geophysical Systems From Any Given Data Sheikholeslami, Razi; Razavi, Saman Geophysical Research Letters, Vol. 47, Issue 20 https://doi.org/10.1029/2020GL089829	journal	October 2020
How much complexity is warranted in a rainfall-runoff model? Jakeman, A. J.; Hornberger, G. M. Water Resources Research, Vol. 29, Issue 8 https://doi.org/10.1029/93WR00877	journal	August 1993
Parameter uncertainty and interaction in complex environmental models Spear, Robert C.; Grieb, Thomas M.; Shang, Nong Water Resources Research, Vol. 30, Issue 11 https://doi.org/10.1029/94WR01732	journal	November 1994
Dilettantism in hydrology: Transition or destiny? Klemeš, V. Water Resources Research, Vol. 22, Issue 9S https://doi.org/10.1029/WR022i09Sp0177S	journal	August 1986
Learning representations by back-propagating errors Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. Nature, Vol. 323, Issue 6088 https://doi.org/10.1038/323533a0	journal	October 1986
Five ways to fix statistics Leek, Jeff; McShane, Blakeley B.; Gelman, Andrew Nature, Vol. 551, Issue 7682 https://doi.org/10.1038/d41586-017-07522-z	journal	November 2017
Five ways to ensure that models serve society: a manifesto Saltelli, Andrea; Bammer, Gabriele; Bruno, Isabelle Nature, Vol. 582, Issue 7813 https://doi.org/10.1038/d41586-020-01812-9	journal	June 2020
Simulating the pandemic: What COVID forecasters can learn from climate models Adam, David Nature, Vol. 587, Issue 7835 https://doi.org/10.1038/d41586-020-03208-1	journal	November 2020
Big names in statistics want to shake up much-maligned P value Singh Chawla, Dalmeet Nature, Vol. 548, Issue 7665 https://doi.org/10.1038/nature.2017.22375	journal	August 2017
Sensitivity of projected long-term CO2 emissions across the Shared Socioeconomic Pathways Marangoni, G.; Tavoni, M.; Bosetti, V. Nature Climate Change, Vol. 7, Issue 2 https://doi.org/10.1038/nclimate3199	journal	January 2017
Practice and perspectives in the validation of resource management models Eker, Sibel; Rovenskaya, Elena; Obersteiner, Michael Nature Communications, Vol. 9, Issue 1 https://doi.org/10.1038/s41467-018-07811-9	journal	December 2018
A short comment on statistical versus mathematical modelling Saltelli, Andrea Nature Communications, Vol. 10, Issue 1 https://doi.org/10.1038/s41467-019-11865-8	journal	August 2019
Robust abatement pathways to tolerable climate futures require immediate global action Lamontagne, J. R.; Reed, P. M.; Marangoni, G. Nature Climate Change, Vol. 9, Issue 4 https://doi.org/10.1038/s41558-019-0426-8	journal	March 2019
Deep learning and process understanding for data-driven Earth system science Reichstein, Markus; Camps-Valls, Gustau; Stevens, Bjorn Nature, Vol. 566, Issue 7743 https://doi.org/10.1038/s41586-019-0912-1	journal	February 2019
Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead Rudin, Cynthia Nature Machine Intelligence, Vol. 1, Issue 5 https://doi.org/10.1038/s42256-019-0048-x	journal	May 2019
From local explanations to global understanding with explainable AI for trees Lundberg, Scott M.; Erion, Gabriel; Chen, Hugh Nature Machine Intelligence, Vol. 2, Issue 1 https://doi.org/10.1038/s42256-019-0138-9	journal	January 2020
The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.18	journal	March 2016
Asymptotic normality and efficiency of two Sobol index estimators Janon, Alexandre; Klein, Thierry; Lagnoux, Agnès ESAIM: Probability and Statistics, Vol. 18 https://doi.org/10.1051/ps/2013040	journal	January 2014
Predicting the Future — Big Data, Machine Learning, and Clinical Medicine Obermeyer, Ziad; Emanuel, Ezekiel J. New England Journal of Medicine, Vol. 375, Issue 13 https://doi.org/10.1056/NEJMp1606181	journal	September 2016
River Stage Forecasting in Bangladesh: Neural Network Approach Liong, Shie-Yui; Lim, Wee-Han; Paudyal, Guna N. Journal of Computing in Civil Engineering, Vol. 14, Issue 1 https://doi.org/10.1061/(ASCE)0887-3801(2000)14:1(1)	journal	January 2000
How Should Robustness Be Defined for Water Systems Planning under Change? Herman, Jonathan D.; Reed, Patrick M.; Zeff, Harrison B. Journal of Water Resources Planning and Management, Vol. 141, Issue 10 https://doi.org/10.1061/(ASCE)WR.1943-5452.0000509	journal	October 2015
Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. I Theory Cukier, R. I.; Fortuin, C. M.; Shuler, K. E. The Journal of Chemical Physics, Vol. 59, Issue 8 https://doi.org/10.1063/1.1680571	journal	October 1973
Improved estimation, using neural networks, of the food consumption of fish populations Lek, S.; Belaud, A.; Dimopoulos, I. Marine and Freshwater Research, Vol. 46, Issue 8 https://doi.org/10.1071/MF9951229	journal	January 1995
An empirical analysis of journal policy effectiveness for computational reproducibility Stodden, Victoria; Seiler, Jennifer; Ma, Zhaokun Proceedings of the National Academy of Sciences, Vol. 115, Issue 11 https://doi.org/10.1073/pnas.1708290115	journal	March 2018
The ASA Statement on p -Values: Context, Process, and Purpose Wasserstein, Ronald L.; Lazar, Nicole A. The American Statistician, Vol. 70, Issue 2 https://doi.org/10.1080/00031305.2016.1154108	journal	April 2016
Advanced Methodology for Uncertainty Propagation in Computer Experiments with Large Number of Inputs Iooss, Bertrand; Marrel, Amandine Nuclear Technology, Vol. 205, Issue 12 https://doi.org/10.1080/00295450.2019.1573617	journal	March 2019
An Analysis for Unreplicated Fractional Factorials Box, George E. P.; Meyer, R. Daniel Technometrics, Vol. 28, Issue 1 https://doi.org/10.1080/00401706.1986.10488093	journal	February 1986
Factorial Sampling Plans for Preliminary Computational Experiments Morris, Max D. Technometrics, Vol. 33, Issue 2 https://doi.org/10.1080/00401706.1991.10484804	journal	May 1991
Density modification-based reliability sensitivity analysis Lemaître, P.; Sergienko, E.; Arnaud, A. Journal of Statistical Computation and Simulation, Vol. 85, Issue 6 https://doi.org/10.1080/00949655.2013.873039	journal	January 2014
Global sensitivity analysis with dependence measures Da Veiga, Sebastien Journal of Statistical Computation and Simulation, Vol. 85, Issue 7 https://doi.org/10.1080/00949655.2014.945932	journal	January 2014
A randomized orthogonal array-based procedure for the estimation of first- and second-order Sobol' indices Tissot, J. -Y.; Prieur, C. Journal of Statistical Computation and Simulation, Vol. 85, Issue 7 https://doi.org/10.1080/00949655.2014.971799	journal	January 2014
New improvements in the use of dependence measures for sensitivity analysis and screening De Lozzo, Matthias; Marrel, Amandine Journal of Statistical Computation and Simulation, Vol. 86, Issue 15 https://doi.org/10.1080/00949655.2016.1149854	journal	February 2016
Sensitivity analysis approaches to high-dimensional screening problems at low sample size Becker, W. E.; Tarantola, S.; Deman, G. Journal of Statistical Computation and Simulation, Vol. 88, Issue 11 https://doi.org/10.1080/00949655.2018.1450876	journal	April 2018
Better Bootstrap Confidence Intervals Efron, Bradley Journal of the American Statistical Association, Vol. 82, Issue 397 https://doi.org/10.1080/01621459.1987.10478410	journal	March 1987
Statistical inference for Sobol pick-freeze Monte Carlo method Gamboa, F.; Janon, A.; Klein, T. Statistics, Vol. 50, Issue 4 https://doi.org/10.1080/02331888.2015.1105803	journal	December 2015
Data-driven modelling approaches for socio-hydrology: opportunities and challenges within the Panta Rhei Science Plan Mount, N. J.; Maier, H. R.; Toth, E. Hydrological Sciences Journal https://doi.org/10.1080/02626667.2016.1159683	journal	April 2016
New sensitivity analysis subordinated to a contrast Fort, Jean-Claude; Klein, Thierry; Rachdi, Nabil Communications in Statistics - Theory and Methods, Vol. 45, Issue 15 https://doi.org/10.1080/03610926.2014.901369	journal	July 2015
Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition Edwards, Marc A.; Roy, Siddhartha Environmental Engineering Science, Vol. 34, Issue 1 https://doi.org/10.1089/ees.2016.0223	journal	January 2017
Sensitivity Analyses for Robust Causal Inference from Mendelian Randomization Analyses with Multiple Genetic Variants Burgess, Stephen; Bowden, Jack; Fall, Tove Epidemiology, Vol. 28, Issue 1 https://doi.org/10.1097/EDE.0000000000000559	journal	January 2017
The natural selection of bad science Smaldino, Paul E.; McElreath, Richard Royal Society Open Science, Vol. 3, Issue 9 https://doi.org/10.1098/rsos.160384	journal	September 2016
Do PISA data justify PISA-based education policy? Araujo, Luisa; Saltelli, Andrea; Schnepf, Sylke V. International Journal of Comparative Education and Development, Vol. 19, Issue 1 https://doi.org/10.1108/IJCED-12-2016-0023	journal	February 2017
Computing as a discipline Denning, P. J.; Comer, D. E.; Gries, D. Computer, Vol. 22, Issue 2 https://doi.org/10.1109/2.19833	journal	February 1989
Global Stock Market Prediction Based on Stock Chart Images Using Deep Q-Network Lee, Jinho; Kim, Raehyun; Koh, Yookyung IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2953542	journal	January 2019
Explainable Machine Learning for Scientific Insights and Discoveries Roscher, Ribana; Bohn, Bastian; Duarte, Marco F. IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.2976199	journal	January 2020
The Software Sustainability Institute: Changing Research Software Attitudes and Practices Crouch, Stephen; Hong, Neil Chue; Hettrick, Simon Computing in Science & Engineering, Vol. 15, Issue 6 https://doi.org/10.1109/MCSE.2013.133	journal	November 2013
Water Science Software Institute: Agile and Open Source Scientific Software Development Ahalt, Stan; Band, Larry; Christopherson, Laura Computing in Science & Engineering, Vol. 16, Issue 3 https://doi.org/10.1109/MCSE.2014.5	journal	May 2014
A Node Pruning Algorithm Based on a Fourier Amplitude Sensitivity Test Method Lauret, P.; Fock, E.; Mara, T. A. IEEE Transactions on Neural Networks, Vol. 17, Issue 2 https://doi.org/10.1109/TNN.2006.871707	journal	March 2006
A New Formulation for Feedforward Neural Networks Razavi, S.; Tolson, B. A. IEEE Transactions on Neural Networks, Vol. 22, Issue 10 https://doi.org/10.1109/TNN.2011.2163169	journal	October 2011
Global Sensitivity Analysis Approach for Input Selection and System Identification Purposes—A New Framework for Feedforward Neural Networks Fock, Eric IEEE Transactions on Neural Networks and Learning Systems, Vol. 25, Issue 8 https://doi.org/10.1109/TNNLS.2013.2294437	journal	August 2014
Global Sensitivity Estimates for Neural Network Classifiers Fernandez-Navarro, Francisco; Carbonero-Ruz, Mariano; Becerra Alonso, David IEEE Transactions on Neural Networks and Learning Systems, Vol. 28, Issue 11 https://doi.org/10.1109/TNNLS.2016.2598657	journal	November 2017
Sensitivity Analysis of k-Fold Cross Validation in Prediction Error Estimation Rodriguez, J. D.; Perez, A.; Lozano, J. A. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, Issue 3 https://doi.org/10.1109/TPAMI.2009.187	journal	March 2010
Determining Inputs for Neural Network Models of Multivariate Time Series Maier, H. R.; Dandy, G. C. Computer-Aided Civil and Infrastructure Engineering, Vol. 12, Issue 5 https://doi.org/10.1111/0885-9507.00069	journal	September 1997
Assumption Hunting in Groundwater Modeling: Find Assumptions Before They Find You: L. Peeters Groundwater xx, no. x: xx-xx Peeters, Luk J. M. Groundwater, Vol. 55, Issue 5 https://doi.org/10.1111/gwat.12565	journal	July 2017
Probabilistic sensitivity analysis of complex models: a Bayesian approach Oakley, Jeremy E.; O'Hagan, Anthony Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3 https://doi.org/10.1111/j.1467-9868.2004.05304.x	journal	August 2004
The Evolution of Scientific Disciplines, Mathematical Modeling, and Human Geography Casetti, Emilio Geographical Analysis, Vol. 31, Issue 4 https://doi.org/10.1111/j.1538-4632.1999.tb00988.x	journal	October 1999
Combining Quantitative and Qualitative Measures of Uncertainty in Model‐Based Environmental Assessment: The NUSAP System Van Der Sluijs, Jeroen P.; Craye, Matthieu; Funtowicz, Silvio Risk Analysis, Vol. 25, Issue 2 https://doi.org/10.1111/j.1539-6924.2005.00604.x	journal	April 2005
Cargo-cult statistics and scientific crisis Stark, Philip B.; Saltelli, Andrea Significance, Vol. 15, Issue 4 https://doi.org/10.1111/j.1740-9713.2018.01174.x	journal	July 2018
Moment-Independent Sensitivity Analysis Using Copula: Moment-Independent Sensitivity Analysis Using Copula Wei, Pengfei; Lu, Zhenzhou; Song, Jingwen Risk Analysis, Vol. 34, Issue 2 https://doi.org/10.1111/risa.12110	journal	September 2013
A Common Rationale for Global Sensitivity Measures and Their Estimation: A Common Rationale for Global Sensitivity Measures Borgonovo, Emanuele; Hazen, Gordon B.; Plischke, Elmar Risk Analysis, Vol. 36, Issue 10 https://doi.org/10.1111/risa.12555	journal	February 2016
Relative Entropy Based Method for Probabilistic Sensitivity Analysis in Engineering Design Liu, Huibin; Chen, Wei; Sudjianto, Agus Journal of Mechanical Design, Vol. 128, Issue 2 https://doi.org/10.1115/1.2159025	journal	January 2006
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play Silver, David; Hubert, Thomas; Schrittwieser, Julian Science, Vol. 362, Issue 6419 https://doi.org/10.1126/science.aar6404	journal	December 2018
A Bayesian Approach for Global Sensitivity Analysis of (Multifidelity) Computer Codes Le Gratiet, Loic; Cannamela, Claire; Iooss, Bertrand SIAM/ASA Journal on Uncertainty Quantification, Vol. 2, Issue 1 https://doi.org/10.1137/130926869	journal	January 2014
Nonlinear Goal-Oriented Bayesian Inference: Application to Carbon Capture and Storage Lieberman, Chad; Willcox, Karen SIAM Journal on Scientific Computing, Vol. 36, Issue 3 https://doi.org/10.1137/130928315	journal	January 2014
Sobol' Indices and Shapley Value Owen, Art B. SIAM/ASA Journal on Uncertainty Quantification, Vol. 2, Issue 1 https://doi.org/10.1137/130936233	journal	January 2014
Optimal Low-rank Approximations of Bayesian Linear Inverse Problems Spantini, Alessio; Solonen, Antti; Cui, Tiangang SIAM Journal on Scientific Computing, Vol. 37, Issue 6 https://doi.org/10.1137/140977308	journal	January 2015
On Shapley Value for Measuring Importance of Dependent Inputs Owen, Art B.; Prieur, Clémentine SIAM/ASA Journal on Uncertainty Quantification, Vol. 5, Issue 1 https://doi.org/10.1137/16M1097717	journal	January 2017
Calibrating a Stochastic, Agent-Based Model Using Quantile-Based Emulation Fadikar, Arindam; Higdon, Dave; Chen, Jiangzhuo SIAM/ASA Journal on Uncertainty Quantification, Vol. 6, Issue 4 https://doi.org/10.1137/17M1161233	journal	January 2018
Global Sensitivity Analysis for Optimization with Variable Selection Spagnol, Adrien; Riche, Rodolphe Le; Veiga, Sébastien Da SIAM/ASA Journal on Uncertainty Quantification, Vol. 7, Issue 2 https://doi.org/10.1137/18M1167978	journal	January 2019
Robustness of the Sobol' Indices to Marginal Distribution Uncertainty Hart, Joseph L.; Gremaud, Pierre A. SIAM/ASA Journal on Uncertainty Quantification, Vol. 7, Issue 4 https://doi.org/10.1137/18M123387X	journal	January 2019
Variance Reduction for Estimation of Shapley Effects and Adaptation to Unknown Input Distribution Broto, Baptiste; Bachoc, François; Depecker, Marine SIAM/ASA Journal on Uncertainty Quantification, Vol. 8, Issue 2 https://doi.org/10.1137/18M1234631	journal	January 2020
The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations Xiu, Dongbin; Karniadakis, George Em SIAM Journal on Scientific Computing, Vol. 24, Issue 2 https://doi.org/10.1137/S1064827501387826	journal	January 2002
ImageNet classification with deep convolutional neural networks Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E. Communications of the ACM, Vol. 60, Issue 6 https://doi.org/10.1145/3065386	journal	May 2017
Dimension Reduction by Local Principal Component Analysis Kambhatla, Nandakishore; Leen, Todd K. Neural Computation, Vol. 9, Issue 7 https://doi.org/10.1162/neco.1997.9.7.1493	journal	October 1997
Exploring Perturbed Physics Ensembles in a Regional Climate Model Bellprat, Omar; Kotlarski, Sven; Lüthi, Daniel Journal of Climate, Vol. 25, Issue 13 https://doi.org/10.1175/JCLI-D-11-00275.1	journal	July 2012
Which Polarimetric Variables Are Important for Weather/No-Weather Discrimination? Lakshmanan, Valliappa; Karstens, Christopher; Krause, John Journal of Atmospheric and Oceanic Technology, Vol. 32, Issue 6 https://doi.org/10.1175/JTECH-D-13-00205.1	journal	June 2015
An Efficient Method for Computing Single-Parameter Partial Expected Value of Perfect Information Strong, Mark; Oakley, Jeremy E. Medical Decision Making, Vol. 33, Issue 6 https://doi.org/10.1177/0272989X12465123	journal	December 2012
Statistical Rituals: The Replication Delusion and How We Got There Gigerenzer, Gerd Advances in Methods and Practices in Psychological Science, Vol. 1, Issue 2 https://doi.org/10.1177/2515245918771329	journal	June 2018
On the Relative Importance of Input Factors in Mathematical Models: Safety Assessment for Nuclear Waste Disposal Saltelli, Andrea; Tarantola, Stefano Journal of the American Statistical Association, Vol. 97, Issue 459 https://doi.org/10.1198/016214502388618447	journal	September 2002
Decision-Theoretic Sensitivity Analysis for Complex Computer Models Oakley, Jeremy E. Technometrics, Vol. 51, Issue 2 https://doi.org/10.1198/TECH.2009.0014	journal	May 2009
Generalized Hoeffding-Sobol decomposition for dependent variables - application to sensitivity analysis Chastaing, Gaelle; Gamboa, Fabrice; Prieur, Clémentine Electronic Journal of Statistics, Vol. 6, Issue 0 https://doi.org/10.1214/12-EJS749	journal	January 2012
Optimum design accounting for the global nonlinear behavior of the model Pázman, Andrej; Pronzato, Luc The Annals of Statistics, Vol. 42, Issue 4 https://doi.org/10.1214/14-AOS1232	journal	August 2014
Poincaré inequalities on intervals – application to sensitivity analysis Roustant, Olivier; Barthe, Franck; Iooss, Bertrand Electronic Journal of Statistics, Vol. 11, Issue 2 https://doi.org/10.1214/17-EJS1310	journal	January 2017
A Class of Statistics with Asymptotically Normal Distribution Hoeffding, Wassily The Annals of Mathematical Statistics, Vol. 19, Issue 3 https://doi.org/10.1214/aoms/1177730196	journal	September 1948
Lattice Sampling Revisited: Monte Carlo Variance of Means Over Randomized Orthogonal Arrays Owen, Art The Annals of Statistics, Vol. 22, Issue 2 https://doi.org/10.1214/aos/1176325504	journal	June 1994
Multivariate Adaptive Regression Splines Friedman, Jerome H. The Annals of Statistics, Vol. 19, Issue 1 https://doi.org/10.1214/aos/1176347963	journal	March 1991
Sensitivity Anaysis as an Ingredient of Modeling Campolongo, F.; Tarantola, S.; Saltelli, A. Statistical Science, Vol. 15, Issue 4 https://doi.org/10.1214/ss/1009213004	journal	November 2000
Design and Analysis of Computer Experiments Sacks, Jerome; Welch, William J.; Mitchell, Toby J. Statistical Science, Vol. 4, Issue 4 https://doi.org/10.1214/ss/1177012413	journal	November 1989
Simulating Sensitivities of Conditional Value at Risk Hong, L. Jeff; Liu, Guangwu Management Science, Vol. 55, Issue 2 https://doi.org/10.1287/mnsc.1080.0901	journal	February 2009
Estimating Quantile Sensitivities Hong, L. Jeff Operations Research, Vol. 57, Issue 1 https://doi.org/10.1287/opre.1080.0531	journal	February 2009
Pathwise Estimation of Probability Sensitivities Through Terminating or Steady-State Simulations Hong, L. Jeff; Liu, Guangwu Operations Research, Vol. 58, Issue 2 https://doi.org/10.1287/opre.1090.0739	journal	April 2010
A Study of Interactions in the Risk Assessment of Complex Engineering Systems: An Application to Space PSA Borgonovo, E.; Smith, C. L. Operations Research, Vol. 59, Issue 6 https://doi.org/10.1287/opre.1110.0973	journal	December 2011
The Value of Information and Stochastic Programming Avriel, M.; Williams, A. C. Operations Research, Vol. 18, Issue 5 https://doi.org/10.1287/opre.18.5.947	journal	October 1970
Faster Kriging: Facing High-Dimensional Simulators Lu, Xuefei; Rudi, Alessandro; Borgonovo, Emanuele Operations Research, Vol. 68, Issue 1 https://doi.org/10.1287/opre.2019.1860	journal	January 2020
Sensitivity Analysis and Performance Extrapolation for Computer Simulation Models Rubinstein, Reuven Y. Operations Research, Vol. 37, Issue 1 https://doi.org/10.1287/opre.37.1.72	journal	February 1989
Improvement of Code Behavior in a Design of Experiments by Metamodeling Bachoc, Francois; Ammar, Karim; Martinez, Jean-Marc Nuclear Science and Engineering, Vol. 183, Issue 3 https://doi.org/10.13182/NSE15-108	journal	July 2016
Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency Kidwell, Mallory C.; Lazarević, Ljiljana B.; Baranski, Erica PLOS Biology, Vol. 14, Issue 5 https://doi.org/10.1371/journal.pbio.1002456	journal	May 2016
Why Most Published Research Findings Are False Ioannidis, John P. A. PLoS Medicine, Vol. 2, Issue 8 https://doi.org/10.1371/journal.pmed.0020124	journal	August 2005
Deep learning for lung cancer prognostication: A retrospective multi-cohort radiomics study Hosny, Ahmed; Parmar, Chintan; Coroller, Thibaud P. PLOS Medicine, Vol. 15, Issue 11 https://doi.org/10.1371/journal.pmed.1002711	journal	November 2018
What do I make of your latinorumc Sensitivity auditing of mathematical modelling Saltelli, Andrea; Guimaraes Pereira, Ângela; Sluijs, Jeroen P. Van der International Journal of Foresight and Innovation Policy, Vol. 9, Issue 2/3/4 https://doi.org/10.1504/IJFIP.2013.058610	journal	January 2013
Plausibility indications in future scenarios Wiek, Arnim; Keeler, Lauren Withycombe; Schweizer, Vanessa International Journal of Foresight and Innovation Policy, Vol. 9, Issue 2/3/4 https://doi.org/10.1504/IJFIP.2013.058611	journal	January 2013
Uncertainties Assessment in Global Sensitivity Indices Estimation from Metamodels Janon, Alexandre; Nodet, Maelle; Prieur, Clementine International Journal for Uncertainty Quantification, Vol. 4, Issue 1 https://doi.org/10.1615/Int.J.UncertaintyQuantification.2012004291	journal	January 2014
Variance-Based Sensitivity Indices of Computer Models with Dependent Inputs: the Fourier Amplitude Sensitivity test Tarantola, S.; Mara, Thierry A. International Journal for Uncertainty Quantification, Vol. 7, Issue 6 https://doi.org/10.1615/Int.J.UncertaintyQuantification.2017020291	journal	January 2017
The Complexities of Agent-Based Modeling Output Analysis Lee, Ju-Sung; Filatova, Tatiana; Ligmann-Zielinska, Arika Journal of Artificial Societies and Social Simulation, Vol. 18, Issue 4 https://doi.org/10.18564/jasss.2897	journal	January 2015
Categorical Inputs, Sensitivity Analysis, Optimization and Importance Tempering with tgp Version 2, an R Package for Treed Gaussian Process Models Gramacy, Robert B.; Taddy, Matthew Journal of Statistical Software, Vol. 33, Issue 6 https://doi.org/10.18637/jss.v033.i06	journal	January 2010
Do rebuttals affect future science? Banobi, Jeannette A.; Branch, Trevor A.; Hilborn, Ray Ecosphere, Vol. 2, Issue 3 https://doi.org/10.1890/ES10-00142.1	journal	March 2011
A general multipurpose interpolation procedure: the magic points Maday, Yvon; Cuong Nguyen, Ngoc; T. Patera, Anthony Communications on Pure & Applied Analysis, Vol. 8, Issue 1 https://doi.org/10.3934/cpaa.2009.8.383	journal	January 2009
Escape from model-land Thompson, Erica L.; Smith, Leonard A. Economics, Vol. 13, Issue 1 https://doi.org/10.5018/economics-ejournal.ja.2019-40	journal	October 2019
What should we do when a model crashes? Recommendations for global sensitivity analysis of Earth and environmental systems models Sheikholeslami, Razi; Razavi, Saman; Haghnegahdar, Amin Geoscientific Model Development, Vol. 12, Issue 10 https://doi.org/10.5194/gmd-12-4275-2019	journal	January 2019
Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation Tang, Y.; Reed, P.; Wagener, T. Hydrology and Earth System Sciences, Vol. 11, Issue 2 https://doi.org/10.5194/hess-11-793-2007	journal	January 2007
Legitimising data-driven models: exemplification of a new data-driven mechanistic modelling framework Mount, N. J.; Dawson, C. W.; Abrahart, R. J. Hydrology and Earth System Sciences, Vol. 17, Issue 7 https://doi.org/10.5194/hess-17-2827-2013	journal	January 2013
Moment-based metrics for global sensitivity analysis of hydrological systems Dell'Oca, Aronne; Riva, Monica; Guadagnini, Alberto Hydrology and Earth System Sciences, Vol. 21, Issue 12 https://doi.org/10.5194/hess-21-6219-2017	journal	January 2017
A randomized orthogonal array-based procedure for the estimation of first- and second-order Sobol' indices Tissot, J. -Y.; Prieur, C. Taylor & Francis https://doi.org/10.6084/m9.figshare.1219361.v2	dataset	January 2015
Sensitivity Analysis in Observational Research: Introducing the E-Value VanderWeele, Tyler J.; Ding, Peng Annals of Internal Medicine, Vol. 167, Issue 4 https://doi.org/10.7326/M16-2607	journal	July 2017

Cited By (1)

Developments and applications of Shapley effects to reliability-oriented sensitivity analysis with correlated inputs Il Idrissi, Marouane; Chabridon, Vincent; Iooss, Bertrand Environmental Modelling & Software, Vol. 143 https://doi.org/10.1016/j.envsoft.2021.105115	journal	September 2021

Similar Records

Environmental studies: discipline or metadiscipline

Journal Article · Fri Dec 31 23:00:00 EST 1982 · Environ. Prof.; (United States) · OSTI ID:7121934

Use of energy policy research and/or analysis in congressional decision making

Thesis/Dissertation · Mon Dec 31 23:00:00 EST 1984 · OSTI ID:7011423

Varying execution discipline to increase performance

Technical Report · Tue Dec 21 23:00:00 EST 1993 · OSTI ID:10131344

Related Subjects

54 ENVIRONMENTAL SCIENCES
Decision Making
Machine Learning
Mathematical Modeling
Model Robustness
Model Validation and Verification
Policy Support
Sensitivity Analysis
Uncertainty Quantification

The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support

Citation Formats

References (261)

Cited By (1)

Similar Records

Related Subjects