DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Simple and statistically sound recommendations for analysing physical theories

Journal Article · · Reports on Progress in Physics
 [1]; ; ; ORCiD logo; ORCiD logo; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more »; ; ; ; ; ; ; ; ; ; ORCiD logo; ; ; ; ; ; ; ORCiD logo; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; « less
  1. University of Minnesota

Physical theories that depend on many parameters or are tested against data from many different experiments pose unique challenges to statistical inference. Many models in particle physics, astrophysics and cosmology fall into one or both of these categories. These issues are often sidestepped with statistically unsound ad hoc methods, involving intersection of parameter intervals estimated by multiple experiments, and random or grid sampling of model parameters. Whilst these methods are easy to apply, they exhibit pathologies even in low-dimensional parameter spaces, and quickly become problematic to use and interpret in higher dimensions. In this article we give clear guidance for going beyond these procedures, suggesting where possible simple methods for performing statistically sound inference, and recommendations of readily-available software tools and standards that can assist in doing so. Our aim is to provide any physicists lacking comprehensive statistical training with recommendations for reaching correct scientific conclusions, with only a modest increase in analysis burden. Furthermore, our examples can be reproduced with the code publicly available at Zenodo.

Research Organization:
Univ. of Minnesota, Minneapolis, MN (United States)
Sponsoring Organization:
USDOE
Contributing Organization:
The GAMBIT Community; The MasterCode Collaboration; The Fittino Collaboration; HEPfit; BayesFits Group
Grant/Contract Number:
SC0011842
OSTI ID:
1905293
Journal Information:
Reports on Progress in Physics, Journal Name: Reports on Progress in Physics Journal Issue: 5 Vol. 85; ISSN 0034-4885
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

References (82)

Author Correction: SciPy 1.0: fundamental algorithms for scientific computing in Python journal February 2020
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence journal March 1987
The Jeffreys–Lindley paradox and discovery criteria in high energy physics journal July 2014
A Simplex Method for Function Minimization journal January 1965
HEPfit: a code for the combination of indirect and direct constraints on high energy physics models journal May 2020
Higgs and supersymmetry journal June 2012
polychord: nested sampling for cosmology journal April 2015
A coverage study of the CMSSM based on ATLAS sensitivity using fast neural networks techniques journal March 2011
An Automatic Method for Finding the Greatest or Least Value of a Function journal March 1960
Use of event-level neutrino telescope data in global fits for theories of new physics journal November 2012
Genetic Algorithms in Astronomy and Astrophysics journal November 1995
dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences journal February 2020
dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences journal February 2020
Optimization by Simulated Annealing journal May 1983
The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters journal February 1971
Impact of vacuum stability, perturbativity and XENON1T on global fits of $$\mathbb {Z}_2$$ Z 2 and $$\mathbb {Z}_3$$ Z 3 scalar singlet dark matter journal October 2018
Matplotlib: A 2D Graphics Environment journal January 2007
The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses journal March 1938
Searching for new phenomena with profile likelihood ratio tests journal April 2020
Fundamental statistical limitations of future dark matter direct detection experiments journal July 2012
Unified approach to the classical statistical analysis of small signals journal April 1998
Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module journal November 2017
Averages of b-hadron, c-hadron, and $$\tau $$-lepton properties as of 2018: Heavy Flavor Averaging Group (HFLAV) journal March 2021
Abandon Statistical Significance journal March 2019
MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics journal October 2009
emcee : The MCMC Hammer
  • Foreman-Mackey, Daniel; Hogg, David W.; Lang, Dustin
  • Publications of the Astronomical Society of the Pacific, Vol. 125, Issue 925 https://doi.org/10.1086/670067
journal March 2013
Review of Particle Physics journal August 2020
Justify your alpha journal February 2018
A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications journal May 2021
Erratum to: Asymptotic formulae for likelihood-based tests of new physics journal July 2013
Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC journal September 2012
Nested sampling for general Bayesian computation journal December 2006
A Statistical Paradox journal January 1957
Prior Probabilities journal January 1968
Global fits of GUT-scale SUSY models with GAMBIT journal December 2017
Dark matter direct-detection experiments journal December 2015
Reporting results in High Energy Physics publications: A manifesto journal November 2020
Bayes Factors journal June 1995
Data Analysis Recipes: Using Markov Chain Monte Carlo journal May 2018
Constrained supersymmetry after two years of LHC data: a global view with Fittino journal June 2012
The Practical Alternative to the p Value Is the Correctly Used p Value journal February 2021
Efficiently Simulating the Coverage Properties of Interval Estimates journal January 1986
Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability journal August 1937
On the Problem of the Most Efficient Tests of Statistical Hypotheses
  • Neyman, J.; Pearson, E. S.
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 231, Issue 694-706 https://doi.org/10.1098/rsta.1933.0009
journal January 1933
Bayes in the sky: Bayesian inference and model selection in cosmology journal March 2008
Comment on “Bayesian Analysis of Pentaquark Signals from CLAS Data” journal July 2008
Asymptotic formulae for likelihood-based tests of new physics journal February 2011
Improved limits on dark matter annihilation in the Sun with the 79-string IceCube detector and implications for supersymmetry journal April 2016
Frequentist evaluation of intervals estimated for a binomial parameter and for the ratio of Poisson means
  • Cousins, Robert D.; Hymes, Kathryn E.; Tucker, Jordan
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 612, Issue 2 https://doi.org/10.1016/j.nima.2009.10.156
journal January 2010
P Values: What They Are and What They Are Not journal August 1996
Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces journal January 1997
GAMBIT: the global and modular beyond-the-standard-model inference tool journal November 2017
MontePython 3: Boosted MCMC sampler and other features journal March 2019
Two cheers for P-values? journal March 2001
A Dirty Dozen: Twelve P-Value Misconceptions journal July 2008
2000 CKM-triangle analysis a critical review with updated experimental inputs and theoretical parameters journal July 2001
A practical solution to the pervasive problems ofp values journal October 2007
Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing journal February 2008
Use of event-level neutrino telescope data in global fits for theories of new physics text January 2012
The fallacy of placing confidence in confidence intervals journal October 2015
The Selection of Prior Distributions by Formal Rules journal September 1996
Statistical coverage for supersymmetric parameter estimation: a case study with direct detection of dark matter journal July 2011
HEPLike: An open source framework for experimental likelihood evaluation journal September 2020
Reproducibility and Replication of Experimental Particle Physics Results journal December 2020
Presentation of search results: the CL s technique journal September 2002
Natural priors, CMSSM fits and LHC weather forecasts journal August 2007
Confusion Over Measures of Evidence ( p 's) Versus Errors ( α 's) in Classical Statistical Testing journal August 2003
Nested sampling for general Bayesian computation journal December 2006
Minuit - a system for function minimization and analysis of the parameter errors and correlations journal December 1975
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations journal April 2016
A Markov chain Monte Carlo analysis of the CMSSM journal May 2006
Planck 2018 results: V. CMB power spectra and likelihoods journal September 2020
The global electroweak fit at NNLO and prospects for the LHC and ILC journal September 2014
Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC journal September 2012
Trial factors for the look elsewhere effect in high energy physics journal October 2010
Constrained MSSM favoring new territories: The impact of new LHC limits and a 125 GeV Higgs boson journal October 2012
Limits and confidence intervals in the presence of nuisance parameters
  • Rolke, Wolfgang A.; López, Angel M.; Conrad, Jan
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 551, Issue 2-3 https://doi.org/10.1016/j.nima.2005.05.068
journal October 2005
Redefine statistical significance journal September 2017
ColliderBit: a GAMBIT module for the calculation of high-energy collider observables and likelihoods journal November 2017
A Short History of Markov Chain Monte Carlo: Subjective Recollections from Incomplete Data journal February 2011
In defense ofPvalues journal March 2014
LikeDM: Likelihood calculator of dark matter detection journal April 2017

Similar Records

WE-A-16A-01: International Medical Physics Symposium: Increasing Access to Medical Physics Education/Training and Research Excellence
Journal Article · 2014 · Medical Physics · OSTI ID:22407910

Veridical data science
Journal Article · 2020 · Proceedings of the National Academy of Sciences of the United States of America · OSTI ID:1625033

Systematically tracking the hourly progression of large wildfires using GOES satellite observations
Journal Article · 2024 · Earth System Science Data (Online) · OSTI ID:2323952

Related Subjects

72