Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events
Abstract
The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementarymore »
- Authors:
-
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Climate and Ecosystem Sciences Division
- Univ. of California, Berkeley, CA (United States). Department of Statistics
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
- Publication Date:
- Research Org.:
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Biological and Environmental Research (BER)
- OSTI Identifier:
- 1477341
- Grant/Contract Number:
- AC02-05CH11231
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of the American Statistical Association
- Additional Journal Information:
- Journal Volume: 113; Journal Issue: 524; Journal ID: ISSN 0162-1459
- Publisher:
- Taylor & Francis
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 54 ENVIRONMENTAL SCIENCES; 97 MATHEMATICS AND COMPUTING; Bayesian nonparametrics; Climate models; Decision theory; Empirical orthogonal functions; Event attribution; False discovery rate; Generalized double Pareto
Citation Formats
Risser, Mark D., Paciorek, Christopher J., and Stone, Dáithí A. Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events. United States: N. p., 2018.
Web. doi:10.1080/01621459.2018.1451335.
Risser, Mark D., Paciorek, Christopher J., & Stone, Dáithí A. Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events. United States. https://doi.org/10.1080/01621459.2018.1451335
Risser, Mark D., Paciorek, Christopher J., and Stone, Dáithí A. Mon .
"Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events". United States. https://doi.org/10.1080/01621459.2018.1451335. https://www.osti.gov/servlets/purl/1477341.
@article{osti_1477341,
title = {Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events},
author = {Risser, Mark D. and Paciorek, Christopher J. and Stone, Dáithí A.},
abstractNote = {The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.},
doi = {10.1080/01621459.2018.1451335},
journal = {Journal of the American Statistical Association},
number = 524,
volume = 113,
place = {United States},
year = {2018},
month = {4}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 6 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
A Bayesian discovery procedure
journal, November 2009
- Guindani, Michele; Müller, Peter; Zhang, Song
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 5
Operating characteristics and extensions of the false discovery rate procedure
journal, August 2002
- Genovese, Christopher; Wasserman, Larry
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 64, Issue 3
On Bayesian Modeling of Fat Tails and Skewness
journal, March 1998
- Fernández, Carmen; Steel, Mark F. J.
- Journal of the American Statistical Association, Vol. 93, Issue 441
Modeling Spatial Variation in Disease Risk: A Geostatistical Approach
journal, September 2002
- Kelsall, Julia; Wakefield, Jonathan
- Journal of the American Statistical Association, Vol. 97, Issue 459
Statistical issues in the analysis of disease mapping data
journal, January 2000
- Pascutto, C.; Wakefield, J. C.; Best, N. G.
- Statistics in Medicine, Vol. 19, Issue 17-18
False Discovery Control for Random Fields
journal, December 2004
- Perone Pacifico, M.; Genovese, C.; Verdinelli, I.
- Journal of the American Statistical Association, Vol. 99, Issue 468
Detecting differential gene expression with a semiparametric hierarchical mixture method
journal, April 2004
- Newton, M. A.
- Biostatistics, Vol. 5, Issue 2
Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution
journal, May 2003
- Azzalini, Adelchi; Capitanio, Antonella
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 65, Issue 2
Selecting the number of components in principal component analysis using cross-validation approximations
journal, June 2012
- Josse, Julie; Husson, François
- Computational Statistics & Data Analysis, Vol. 56, Issue 6
The positive false discovery rate: a Bayesian interpretation and the q -value
journal, December 2003
- Storey, John D.
- The Annals of Statistics, Vol. 31, Issue 6
Multiple testing for neuroimaging via hidden Markov random field: Multiple Testing for Neuroimaging via Hidden Markov Random Field
journal, May 2015
- Shu, Hai; Nan, Bin; Koeppe, Robert
- Biometrics, Vol. 71, Issue 3
On the Attribution of a Single Event to Climate Change
journal, November 2014
- Hansen, Gerrit; Auffhammer, Maximilian; Solow, Andrew R.
- Journal of Climate, Vol. 27, Issue 22
False discovery control in large-scale spatial multiple testing
journal, April 2014
- Sun, Wenguang; Reich, Brian J.; Tony Cai, T.
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 77, Issue 1
“The Stippling Shows Statistically Significant Grid Points”: How Research Results are Routinely Overstated and Overinterpreted, and What to Do about It
journal, December 2016
- Wilks, D. S.
- Bulletin of the American Meteorological Society, Vol. 97, Issue 12
Empirical Bayes Analysis of a Microarray Experiment
journal, December 2001
- Efron, Bradley; Tibshirani, Robert; Storey, John D.
- Journal of the American Statistical Association, Vol. 96, Issue 456
Bayesian principal component regression with data-driven component selection
journal, June 2012
- Wang, Liuxia
- Journal of Applied Statistics, Vol. 39, Issue 6
Bayesian image restoration, with two applications in spatial statistics
journal, March 1991
- Besag, Julian; York, Jeremy; Molli�, Annie
- Annals of the Institute of Statistical Mathematics, Vol. 43, Issue 1
Anthropogenic greenhouse gas contribution to flood risk in England and Wales in autumn 2000
journal, February 2011
- Pall, Pardeep; Aina, Tolu; Stone, Dáithí A.
- Nature, Vol. 470, Issue 7334
Dimension reduction and alleviation of confounding for spatial generalized linear mixed models
journal, October 2012
- Hughes, John; Haran, Murali
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 75, Issue 1
Linear Models for Airborne-Laser-Scanning-Based Operational Forest Inventory With Small Field Sample Size and Highly Correlated LiDAR Data
journal, October 2015
- Junttila, Virpi; Kauranne, Tuomo; Finley, Andrew O.
- IEEE Transactions on Geoscience and Remote Sensing, Vol. 53, Issue 10
Large-scale multiple testing under dependence
journal, April 2009
- Sun, Wenguang; Tony Cai, T.
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 2
An Independent Assessment of Anthropogenic Attribution Statements for Recent Extreme Temperature and Rainfall Events
journal, January 2017
- Angélil, Oliver; Stone, Dáithí; Wehner, Michael
- Journal of Climate, Vol. 30, Issue 1
Bayesian inference for finite mixtures of univariate and multivariate skew-normal and skew-t distributions
journal, January 2010
- Fruhwirth-Schnatter, S.; Pyne, S.
- Biostatistics, Vol. 11, Issue 2
On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics
journal, March 2000
- Benjamini, Yoav; Hochberg, Yosef
- Journal of Educational and Behavioral Statistics, Vol. 25, Issue 1
Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk
journal, December 1990
- Farrington, Conor P.; Manning, Godfrey
- Statistics in Medicine, Vol. 9, Issue 12
Covariance regularization by thresholding
journal, December 2008
- Bickel, Peter J.; Levina, Elizaveta
- The Annals of Statistics, Vol. 36, Issue 6
Attribution of extreme weather to anthropogenic greenhouse gas emissions: Sensitivity to spatial and temporal scales
journal, March 2014
- Angélil, Oliver; Stone, Dáithí A.; Tadross, Mark
- Geophysical Research Letters, Vol. 41, Issue 6
Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control
journal, September 2007
- Sun, Wenguang; Cai, T. Tony
- Journal of the American Statistical Association, Vol. 102, Issue 479
Comparing regional precipitation and temperature extremes in climate model and reanalysis products
journal, September 2016
- Angélil, Oliver; Perkins-Kirkpatrick, Sarah; Alexander, Lisa V.
- Weather and Climate Extremes, Vol. 13
Bayesian regression based on principal components for high-dimensional data
journal, May 2013
- Lee, Jaeyong; Oh, Hee-Seok
- Journal of Multivariate Analysis, Vol. 117
under dependency
journal, August 2001
- Yekutieli, Daniel; Benjamini, Yoav
- The Annals of Statistics, Vol. 29, Issue 4
The End-to-End Attribution Problem: From Emissions to Impacts
journal, August 2005
- Stone, Dáithí A.; Allen, Myles R.
- Climatic Change, Vol. 71, Issue 3
The horseshoe estimator for sparse signals
journal, April 2010
- Carvalho, C. M.; Polson, N. G.; Scott, J. G.
- Biometrika, Vol. 97, Issue 2
The Scree Test For The Number Of Factors
journal, April 1966
- Cattell, Raymond B.
- Multivariate Behavioral Research, Vol. 1, Issue 2
The Late Onset of the 2015 Wet Season in Nigeria
journal, December 2016
- Lawal, Kamoru A.; Abatan, Abayomi A.; Angélil, Oliver
- Bulletin of the American Meteorological Society, Vol. 97, Issue 12
A Bayesian hierarchical model for climate change detection and attribution: BAYESIAN DETECTION AND ATTRIBUTION
journal, June 2017
- Katzfuss, Matthias; Hammerling, Dorit; Smith, Richard L.
- Geophysical Research Letters, Vol. 44, Issue 11
Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE
journal, October 2016
- de Valpine, Perry; Turek, Daniel; Paciorek, Christopher J.
- Journal of Computational and Graphical Statistics, Vol. 26, Issue 2
Multinomial Inverse Regression for Text Analysis
journal, September 2013
- Taddy, Matt
- Journal of the American Statistical Association, Vol. 108, Issue 503
A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5.1-1degree
journal, March 2018
- Stone, Dáithí A.; Risser, Mark D.; Angélil, Oliver M.
- Weather and Climate Extremes, Vol. 19
Optimal Sample Size for Multiple Testing: The Case of Gene Expression Microarrays
journal, December 2004
- Müller, Peter; Parmigiani, Giovanni; Robert, Christian
- Journal of the American Statistical Association, Vol. 99, Issue 468
The centred parametrization for the multivariate skew-normal distribution
journal, August 2008
- Arellano-Valle, Reinaldo B.; Azzalini, Adelchi
- Journal of Multivariate Analysis, Vol. 99, Issue 7
Cross-Validatory Estimation of the Number of Components in Factor and Principal Components Models
journal, November 1978
- Wold, Svante
- Technometrics, Vol. 20, Issue 4
Good, great, or lucky? Screening for firms with sustained superior performance using heavy-tailed priors
journal, March 2012
- Polson, Nicholas G.; Scott, James G.
- The Annals of Applied Statistics, Vol. 6, Issue 1
Shrinkage Estimators for Covariance Matrices
journal, December 2001
- Daniels, Michael J.; Kass, Robert E.
- Biometrics, Vol. 57, Issue 4
Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data
journal, November 2004
- Ventura, Valérie; Paciorek, Christopher J.; Risbey, James S.
- Journal of Climate, Vol. 17, Issue 22
Multiscale Spatial Density Smoothing: An Application to Large-Scale Radiological Survey and Anomaly Detection
journal, July 2016
- Tansey, Wesley; Athey, Alex; Reinhart, Alex
- Journal of the American Statistical Association, Vol. 112, Issue 519
Statistical Field Significance and its Determination by Monte Carlo Techniques
journal, January 1983
- Livezey, Robert E.; Chen, W. Y.
- Monthly Weather Review, Vol. 111, Issue 1
Human contribution to the European heatwave of 2003
journal, December 2004
- Stott, Peter A.; Stone, D. A.; Allen, M. R.
- Nature, Vol. 432, Issue 7017
Size, power and false discovery rates
journal, August 2007
- Efron, Bradley
- The Annals of Statistics, Vol. 35, Issue 4
A Computationally Efficient Projection-Based Approach for Spatial Generalized Linear Mixed Models
journal, October 2018
- Guan, Yawen; Haran, Murali
- Journal of Computational and Graphical Statistics, Vol. 27, Issue 4
Lethal methemoglobinemia in the invasive brown treesnake after acetaminophen ingestion
journal, January 2020
- Mathies, Tom; Mauldin, Richard E.
- Scientific Reports, Vol. 10, Issue 1
Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data
text, January 2012
- Ventura, Valerie; Paciorek, Christopher J.; Risbey, James S.
- Carnegie Mellon University
Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data
text, January 2012
- Ventura, Valerie; Paciorek, Christopher J.; Risbey, James S.
- Carnegie Mellon University
On Bayesian Modeling of Fat Tails and Skewness
journal, March 1998
- Fernandez, Carmen; Steel, Mark F. J.
- Journal of the American Statistical Association, Vol. 93, Issue 441
On the Adaptive Control of the False Discovery Rate in Multiple Testing with Independent Statistics
journal, April 2000
- Benjamini, Yoav; Hochberg, Yosef
- Journal of Educational and Behavioral Statistics, Vol. 25, Issue 1
Good, great, or lucky? Screening for firms with sustained superior performance using heavy-tailed priors
text, January 2010
- Polson, Nicholas G.; Scott, James G.
- arXiv
Programming with models: writing statistical algorithms for general model structures with NIMBLE
text, January 2015
- de Valpine, Perry; Turek, Daniel; Paciorek, Christopher J.
- arXiv