Statistical Significance Testing for Mixed Priors: A Combined Bayesian and Frequentist Analysis

Journal Article · Wed Sep 21 00:00:00 EDT 2022 · Entropy

DOI:https://doi.org/10.3390/e24101328· OSTI ID:1994364

Robnik, Jakob ^[1]; Seljak, Uroš ^[2]

---

1. University of California, Berkeley, CA (United States)
2. University of California, Berkeley, CA (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

In many hypothesis testing applications, we have mixed priors, with well-motivated informative priors for some parameters but not for others. The Bayesian methodology uses the Bayes factor and is helpful for the informative priors, as it incorporates Occam’s razor via the multiplicity or trials factor in the look-elsewhere effect. However, if the prior is not known completely, the frequentist hypothesis test via the false-positive rate is a better approach, as it is less sensitive to the prior choice. We argue that when only partial prior information is available, it is best to combine the two methodologies by using the Bayes factor as a test statistic in the frequentist analysis. We show that the standard frequentist maximum likelihood-ratio test statistic corresponds to the Bayes factor with a non-informative Jeffrey’s prior. We also show that mixed priors increase the statistical power in frequentist analyses over the maximum likelihood test statistic. We develop an analytic formalism that does not require expensive simulations and generalize Wilks’ theorem beyond its usual regime of validity. In specific limits, the formalism reproduces existing expressions, such as the p-value of linear models and periodograms. We apply the formalism to an example of exoplanet transits, where multiplicity can be more than 10⁷. We show that our analytic expressions reproduce the $$p$$-values derived from numerical simulations. We offer an interpretation of our formalism based on the statistical mechanics. We introduce the counting of states in a continuous parameter space using the uncertainty volume as the quantum of the state. We show that both the $$p$$-value and Bayes factor can be expressed as an energy versus entropy competition.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: Heising-Simons Foundation; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• Grant/Contract Number: AC02-05CH11231
• OSTI ID: 1994364
• Journal Information: Entropy, Journal Name: Entropy Journal Issue: 10 Vol. 24; ISSN 1099-4300
• Publisher: MDPI
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian statistics
exoplanet transit search
frequentist statistics
hypothesis testing
look-elsewhere effect
statistical mechanics