skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

Abstract

The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementarymore » materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.« less

Authors:
 [1];  [2];  [3]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Climate and Ecosystem Sciences Division
  2. Univ. of California, Berkeley, CA (United States). Department of Statistics
  3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)
OSTI Identifier:
1477341
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Journal of the American Statistical Association
Additional Journal Information:
Journal Volume: 113; Journal Issue: 524; Journal ID: ISSN 0162-1459
Publisher:
Taylor & Francis
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; 97 MATHEMATICS AND COMPUTING; Bayesian nonparametrics; Climate models; Decision theory; Empirical orthogonal functions; Event attribution; False discovery rate; Generalized double Pareto

Citation Formats

Risser, Mark D., Paciorek, Christopher J., and Stone, Dáithí A. Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events. United States: N. p., 2018. Web. doi:10.1080/01621459.2018.1451335.
Risser, Mark D., Paciorek, Christopher J., & Stone, Dáithí A. Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events. United States. doi:10.1080/01621459.2018.1451335.
Risser, Mark D., Paciorek, Christopher J., and Stone, Dáithí A. Mon . "Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events". United States. doi:10.1080/01621459.2018.1451335. https://www.osti.gov/servlets/purl/1477341.
@article{osti_1477341,
title = {Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events},
author = {Risser, Mark D. and Paciorek, Christopher J. and Stone, Dáithí A.},
abstractNote = {The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.},
doi = {10.1080/01621459.2018.1451335},
journal = {Journal of the American Statistical Association},
number = 524,
volume = 113,
place = {United States},
year = {2018},
month = {4}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 4 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

A Bayesian discovery procedure
journal, November 2009

  • Guindani, Michele; Müller, Peter; Zhang, Song
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 5
  • DOI: 10.1111/j.1467-9868.2009.00714.x

Operating characteristics and extensions of the false discovery rate procedure
journal, August 2002

  • Genovese, Christopher; Wasserman, Larry
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 64, Issue 3
  • DOI: 10.1111/1467-9868.00347

Modeling Spatial Variation in Disease Risk: A Geostatistical Approach
journal, September 2002

  • Kelsall, Julia; Wakefield, Jonathan
  • Journal of the American Statistical Association, Vol. 97, Issue 459
  • DOI: 10.1198/016214502388618438

False Discovery Control for Random Fields
journal, December 2004

  • Perone Pacifico, M.; Genovese, C.; Verdinelli, I.
  • Journal of the American Statistical Association, Vol. 99, Issue 468
  • DOI: 10.1198/0162145000001655

Empirical Bayes Analysis of a Microarray Experiment
journal, December 2001

  • Efron, Bradley; Tibshirani, Robert; Storey, John D.
  • Journal of the American Statistical Association, Vol. 96, Issue 456
  • DOI: 10.1198/016214501753382129

Anthropogenic greenhouse gas contribution to flood risk in England and Wales in autumn 2000
journal, February 2011

  • Pall, Pardeep; Aina, Tolu; Stone, Dáithí A.
  • Nature, Vol. 470, Issue 7334
  • DOI: 10.1038/nature09762

Large-scale multiple testing under dependence
journal, April 2009

  • Sun, Wenguang; Tony Cai, T.
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 2
  • DOI: 10.1111/j.1467-9868.2008.00694.x

An Independent Assessment of Anthropogenic Attribution Statements for Recent Extreme Temperature and Rainfall Events
journal, January 2017


Bayesian inference for finite mixtures of univariate and multivariate skew-normal and skew-t distributions
journal, January 2010


On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics
journal, March 2000

  • Benjamini, Yoav; Hochberg, Yosef
  • Journal of Educational and Behavioral Statistics, Vol. 25, Issue 1
  • DOI: 10.3102/10769986025001060

Liability for climate change
journal, February 2003


Covariance regularization by thresholding
journal, December 2008

  • Bickel, Peter J.; Levina, Elizaveta
  • The Annals of Statistics, Vol. 36, Issue 6
  • DOI: 10.1214/08-AOS600

Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control
journal, September 2007

  • Sun, Wenguang; Cai, T. Tony
  • Journal of the American Statistical Association, Vol. 102, Issue 479
  • DOI: 10.1198/016214507000000545

The End-to-End Attribution Problem: From Emissions to Impacts
journal, August 2005


The horseshoe estimator for sparse signals
journal, April 2010


A Bayesian hierarchical model for climate change detection and attribution: BAYESIAN DETECTION AND ATTRIBUTION
journal, June 2017

  • Katzfuss, Matthias; Hammerling, Dorit; Smith, Richard L.
  • Geophysical Research Letters, Vol. 44, Issue 11
  • DOI: 10.1002/2017GL073688

A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5.1-1degree
journal, March 2018


Optimal Sample Size for Multiple Testing: The Case of Gene Expression Microarrays
journal, December 2004

  • Müller, Peter; Parmigiani, Giovanni; Robert, Christian
  • Journal of the American Statistical Association, Vol. 99, Issue 468
  • DOI: 10.1198/016214504000001646

Good, great, or lucky? Screening for firms with sustained superior performance using heavy-tailed priors
journal, March 2012

  • Polson, Nicholas G.; Scott, James G.
  • The Annals of Applied Statistics, Vol. 6, Issue 1
  • DOI: 10.1214/11-AOAS512

Multiscale Spatial Density Smoothing: An Application to Large-Scale Radiological Survey and Anomaly Detection
journal, July 2016

  • Tansey, Wesley; Athey, Alex; Reinhart, Alex
  • Journal of the American Statistical Association, Vol. 112, Issue 519
  • DOI: 10.1080/01621459.2016.1276461

Human contribution to the European heatwave of 2003
journal, December 2004

  • Stott, Peter A.; Stone, D. A.; Allen, M. R.
  • Nature, Vol. 432, Issue 7017
  • DOI: 10.1038/nature03089

On Bayesian Modeling of Fat Tails and Skewness
journal, March 1998


Statistical issues in the analysis of disease mapping data
journal, January 2000


Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution
journal, May 2003

  • Azzalini, Adelchi; Capitanio, Antonella
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 65, Issue 2
  • DOI: 10.1111/1467-9868.00391

Selecting the number of components in principal component analysis using cross-validation approximations
journal, June 2012


The positive false discovery rate: a Bayesian interpretation and the q -value
journal, December 2003


On the Attribution of a Single Event to Climate Change
journal, November 2014

  • Hansen, Gerrit; Auffhammer, Maximilian; Solow, Andrew R.
  • Journal of Climate, Vol. 27, Issue 22
  • DOI: 10.1175/JCLI-D-14-00399.1

False discovery control in large-scale spatial multiple testing
journal, April 2014

  • Sun, Wenguang; Reich, Brian J.; Tony Cai, T.
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 77, Issue 1
  • DOI: 10.1111/rssb.12064

Bayesian principal component regression with data-driven component selection
journal, June 2012


Bayesian image restoration, with two applications in spatial statistics
journal, March 1991

  • Besag, Julian; York, Jeremy; Molli�, Annie
  • Annals of the Institute of Statistical Mathematics, Vol. 43, Issue 1
  • DOI: 10.1007/BF00116466

Dimension reduction and alleviation of confounding for spatial generalized linear mixed models
journal, October 2012

  • Hughes, John; Haran, Murali
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 75, Issue 1
  • DOI: 10.1111/j.1467-9868.2012.01041.x

Linear Models for Airborne-Laser-Scanning-Based Operational Forest Inventory With Small Field Sample Size and Highly Correlated LiDAR Data
journal, October 2015

  • Junttila, Virpi; Kauranne, Tuomo; Finley, Andrew O.
  • IEEE Transactions on Geoscience and Remote Sensing, Vol. 53, Issue 10
  • DOI: 10.1109/TGRS.2015.2425916

Attribution of extreme weather to anthropogenic greenhouse gas emissions: Sensitivity to spatial and temporal scales
journal, March 2014

  • Angélil, Oliver; Stone, Dáithí A.; Tadross, Mark
  • Geophysical Research Letters, Vol. 41, Issue 6
  • DOI: 10.1002/2014GL059234

Comparing regional precipitation and temperature extremes in climate model and reanalysis products
journal, September 2016

  • Angélil, Oliver; Perkins-Kirkpatrick, Sarah; Alexander, Lisa V.
  • Weather and Climate Extremes, Vol. 13
  • DOI: 10.1016/j.wace.2016.07.001

Bayesian regression based on principal components for high-dimensional data
journal, May 2013


under dependency
journal, August 2001


The Scree Test For The Number Of Factors
journal, April 1966


The Late Onset of the 2015 Wet Season in Nigeria
journal, December 2016

  • Lawal, Kamoru A.; Abatan, Abayomi A.; Angélil, Oliver
  • Bulletin of the American Meteorological Society, Vol. 97, Issue 12
  • DOI: 10.1175/BAMS-D-16-0131.1

Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE
journal, October 2016

  • de Valpine, Perry; Turek, Daniel; Paciorek, Christopher J.
  • Journal of Computational and Graphical Statistics, Vol. 26, Issue 2
  • DOI: 10.1080/10618600.2016.1172487

Multinomial Inverse Regression for Text Analysis
journal, September 2013


The centred parametrization for the multivariate skew-normal distribution
journal, August 2008

  • Arellano-Valle, Reinaldo B.; Azzalini, Adelchi
  • Journal of Multivariate Analysis, Vol. 99, Issue 7
  • DOI: 10.1016/j.jmva.2008.01.020

Shrinkage Estimators for Covariance Matrices
journal, December 2001


Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data
journal, November 2004

  • Ventura, Valérie; Paciorek, Christopher J.; Risbey, James S.
  • Journal of Climate, Vol. 17, Issue 22
  • DOI: 10.1175/3199.1

Statistical Field Significance and its Determination by Monte Carlo Techniques
journal, January 1983


Cross-Validatory Estimation of the Number of Components in Factor and Principal Components Models
journal, November 1978


    Works referencing / citing this record:

    Bayesian principal component regression with data-driven component selection
    journal, June 2012


    On Bayesian Modeling of Fat Tails and Skewness
    journal, March 1998


    Liability for climate change
    journal, February 2003


    Bayesian inference for finite mixtures of univariate and multivariate skew-normal and skew-t distributions
    journal, January 2010


    Large-scale multiple testing under dependence
    journal, April 2009

    • Sun, Wenguang; Tony Cai, T.
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 2
    • DOI: 10.1111/j.1467-9868.2008.00694.x

    Selecting the number of components in principal component analysis using cross-validation approximations
    journal, June 2012


    Optimal Sample Size for Multiple Testing: The Case of Gene Expression Microarrays
    journal, December 2004

    • Müller, Peter; Parmigiani, Giovanni; Robert, Christian
    • Journal of the American Statistical Association, Vol. 99, Issue 468
    • DOI: 10.1198/016214504000001646

    The positive false discovery rate: a Bayesian interpretation and the q -value
    journal, December 2003


    Controlling the Proportion of Falsely Rejected Hypotheses when Conducting Multiple Tests with Climatological Data
    journal, November 2004

    • Ventura, Valérie; Paciorek, Christopher J.; Risbey, James S.
    • Journal of Climate, Vol. 17, Issue 22
    • DOI: 10.1175/3199.1

    The horseshoe estimator for sparse signals
    journal, April 2010


    False discovery control in large-scale spatial multiple testing
    journal, April 2014

    • Sun, Wenguang; Reich, Brian J.; Tony Cai, T.
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 77, Issue 1
    • DOI: 10.1111/rssb.12064

    Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE
    journal, October 2016

    • de Valpine, Perry; Turek, Daniel; Paciorek, Christopher J.
    • Journal of Computational and Graphical Statistics, Vol. 26, Issue 2
    • DOI: 10.1080/10618600.2016.1172487

    Operating characteristics and extensions of the false discovery rate procedure
    journal, August 2002

    • Genovese, Christopher; Wasserman, Larry
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 64, Issue 3
    • DOI: 10.1111/1467-9868.00347

    Bayesian regression based on principal components for high-dimensional data
    journal, May 2013


    Comparing regional precipitation and temperature extremes in climate model and reanalysis products
    journal, September 2016

    • Angélil, Oliver; Perkins-Kirkpatrick, Sarah; Alexander, Lisa V.
    • Weather and Climate Extremes, Vol. 13
    • DOI: 10.1016/j.wace.2016.07.001

    The centred parametrization for the multivariate skew-normal distribution
    journal, August 2008

    • Arellano-Valle, Reinaldo B.; Azzalini, Adelchi
    • Journal of Multivariate Analysis, Vol. 99, Issue 7
    • DOI: 10.1016/j.jmva.2008.01.020

    Anthropogenic greenhouse gas contribution to flood risk in England and Wales in autumn 2000
    journal, February 2011

    • Pall, Pardeep; Aina, Tolu; Stone, Dáithí A.
    • Nature, Vol. 470, Issue 7334
    • DOI: 10.1038/nature09762

    Multiscale Spatial Density Smoothing: An Application to Large-Scale Radiological Survey and Anomaly Detection
    journal, July 2016

    • Tansey, Wesley; Athey, Alex; Reinhart, Alex
    • Journal of the American Statistical Association, Vol. 112, Issue 519
    • DOI: 10.1080/01621459.2016.1276461

    Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control
    journal, September 2007

    • Sun, Wenguang; Cai, T. Tony
    • Journal of the American Statistical Association, Vol. 102, Issue 479
    • DOI: 10.1198/016214507000000545

    False Discovery Control for Random Fields
    journal, December 2004

    • Perone Pacifico, M.; Genovese, C.; Verdinelli, I.
    • Journal of the American Statistical Association, Vol. 99, Issue 468
    • DOI: 10.1198/0162145000001655

    Human contribution to the European heatwave of 2003
    journal, December 2004

    • Stott, Peter A.; Stone, D. A.; Allen, M. R.
    • Nature, Vol. 432, Issue 7017
    • DOI: 10.1038/nature03089

    The Scree Test For The Number Of Factors
    journal, April 1966


    under dependency
    journal, August 2001


    Multinomial Inverse Regression for Text Analysis
    journal, September 2013


    The End-to-End Attribution Problem: From Emissions to Impacts
    journal, August 2005


    A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5.1-1degree
    journal, March 2018


    A Bayesian discovery procedure
    journal, November 2009

    • Guindani, Michele; Müller, Peter; Zhang, Song
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 5
    • DOI: 10.1111/j.1467-9868.2009.00714.x

    Modeling Spatial Variation in Disease Risk: A Geostatistical Approach
    journal, September 2002

    • Kelsall, Julia; Wakefield, Jonathan
    • Journal of the American Statistical Association, Vol. 97, Issue 459
    • DOI: 10.1198/016214502388618438

    Distributions generated by perturbation of symmetry with emphasis on a multivariate skew t-distribution
    journal, May 2003

    • Azzalini, Adelchi; Capitanio, Antonella
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 65, Issue 2
    • DOI: 10.1111/1467-9868.00391

    Dimension reduction and alleviation of confounding for spatial generalized linear mixed models
    journal, October 2012

    • Hughes, John; Haran, Murali
    • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 75, Issue 1
    • DOI: 10.1111/j.1467-9868.2012.01041.x

    On the Adaptive Control of the False Discovery Rate in Multiple Testing With Independent Statistics
    journal, March 2000

    • Benjamini, Yoav; Hochberg, Yosef
    • Journal of Educational and Behavioral Statistics, Vol. 25, Issue 1
    • DOI: 10.3102/10769986025001060

    Empirical Bayes Analysis of a Microarray Experiment
    journal, December 2001

    • Efron, Bradley; Tibshirani, Robert; Storey, John D.
    • Journal of the American Statistical Association, Vol. 96, Issue 456
    • DOI: 10.1198/016214501753382129