Simple and statistically sound recommendations for analysing physical theories

AbdusSalam, Shehu S.; Agocs, Fruzsina J.; Allanach, Benjamin C.; Athron, Peter; Balázs, Csaba; Bagnaschi, Emanuele; Bechtle, Philip; Buchmueller, Oliver; Beniwal, Ankit; Bhom, Jihyun; Bloor, Sanjay; Bringmann, Torsten; Buckley, Andy; Butter, Anja; Camargo-Molina, José Eliel; Chrzaszcz, Marcin; Conrad, Jan; Cornell, Jonathan M.; Danninger, Matthias; de Blas, Jorge; De Roeck, Albert; Desch, Klaus; Dolan, Matthew; Dreiner, Herbert; Eberhardt, Otto; Ellis, John; Farmer, Ben; Fedele, Marco; Flächer, Henning; Fowlie, Andrew; Gonzalo, Tomás E.; Grace, Philip; Hamer, Matthias; Handley, Will; Harz, Julia; Heinemeyer, Sven; Hoof, Sebastian; Hotinli, Selim; Jackson, Paul; Kahlhoefer, Felix; Kowalska, Kamila; Krämer, Michael; Kvellestad, Anders; Martinez, Miriam Lucio; Mahmoudi, Farvah; Santos, Diego Martinez; Martinez, Gregory D.; Mishima, Satoshi; Olive, Keith; Paul, Ayan; Prim, Markus Tobias; Porod, Werner; Raklev, Are; Renk, Janina J.; Rogan, Christopher; Roszkowski, Leszek; Ruiz de Austri, Roberto; Sakurai, Kazuki; Scaffidi, Andre; Scott, Pat; Sessolo, Enrico Maria; Stefaniak, Tim; Stöcker, Patrick; Su, Wei; Trojanowski, Sebastian; Trotta, Roberto; Sming Tsai, Yue-Lin; Van den Abeele, Jeriek; Valli, Mauro; Vincent, Aaron C.; Weiglein, Georg; White, Martin; Wienemann, Peter; Wu, Lei; Zhang, Yang

doi:10.1088/1361-6633/ac60ac

Title: Simple and statistically sound recommendations for analysing physical theories

Journal Article · 29 April 2022 · Reports on Progress in Physics

DOI: https://doi.org/10.1088/1361-6633/ac60ac · OSTI ID:1905293

AbdusSalam, Shehu S.; Agocs, Fruzsina J.; Allanach, Benjamin C.;

;

; Bagnaschi, Emanuele; Bechtle, Philip; Buchmueller, Oliver; Beniwal, Ankit; Bhom, Jihyun; Bloor, Sanjay; Bringmann, Torsten; Buckley, Andy; Butter, Anja; Camargo-Molina, José Eliel; Chrzaszcz, Marcin; Conrad, Jan; Cornell, Jonathan M.; Danninger, Matthias; de Blas, Jorge more »

Physical theories that depend on many parameters or are tested against data from many different experiments pose unique challenges to statistical inference. Many models in particle physics, astrophysics and cosmology fall into one or both of these categories. These issues are often sidestepped with statistically unsound ad hoc methods, involving intersection of parameter intervals estimated by multiple experiments, and random or grid sampling of model parameters. Whilst these methods are easy to apply, they exhibit pathologies even in low-dimensional parameter spaces, and quickly become problematic to use and interpret in higher dimensions. In this article we give clear guidance for going beyond these procedures, suggesting where possible simple methods for performing statistically sound inference, and recommendations of readily-available software tools and standards that can assist in doing so. Our aim is to provide any physicists lacking comprehensive statistical training with recommendations for reaching correct scientific conclusions, with only a modest increase in analysis burden. Furthermore, our examples can be reproduced with the code publicly available at Zenodo.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Univ. of Minnesota, Minneapolis, MN (United States)

Sponsoring Organization:: USDOE

Contributing Organization:: The GAMBIT Community; The MasterCode Collaboration; The Fittino Collaboration; HEPfit; BayesFits Group

Grant/Contract Number:: SC0011842

OSTI ID:: 1905293

Journal Information:: Reports on Progress in Physics, Vol. 85, Issue 5; ISSN 0034-4885

Publisher:: IOP PublishingCopyright Statement

Country of Publication:: United States

Language:: English

References (70)

Bayes Factors Kass, Robert E.; Raftery, Adrian E. Journal of the American Statistical Association, Vol. 90, Issue 430 https://doi.org/10.1080/01621459.1995.10476572	journal	June 1995
Impact of vacuum stability, perturbativity and XENON1T on global fits of $$\mathbb {Z}_2$$ Z 2 and $$\mathbb {Z}_3$$ Z 3 scalar singlet dark matter Athron, Peter; Cornell, Jonathan M.; Kahlhoefer, Felix The European Physical Journal C, Vol. 78, Issue 10 https://doi.org/10.1140/epjc/s10052-018-6314-y	journal	October 2018
Use of event-level neutrino telescope data in global fits for theories of new physics Scott, P.; Savage, C.; Edsjö, J. arXiv https://doi.org/10.48550/arxiv.1207.0810	text	January 2012
Global fits of GUT-scale SUSY models with GAMBIT Athron, Peter; Balázs, Csaba; Bringmann, Torsten The European Physical Journal C, Vol. 77, Issue 12 https://doi.org/10.1140/epjc/s10052-017-5167-0	journal	December 2017
On the Problem of the Most Efficient Tests of Statistical Hypotheses Neyman, J.; Pearson, E. S. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 231, Issue 694-706 https://doi.org/10.1098/rsta.1933.0009	journal	January 1933
P Values: What They Are and What They Are Not Schervish, Mark J. The American Statistician, Vol. 50, Issue 3 https://doi.org/10.2307/2684655	journal	August 1996
Averages of b-hadron, c-hadron, and $$\tau $$-lepton properties as of 2018: Heavy Flavor Averaging Group (HFLAV) Amhis, Y.; Banerjee, Sw.; Ben-Haim, E. The European Physical Journal C, Vol. 81, Issue 3 https://doi.org/10.1140/epjc/s10052-020-8156-7	journal	March 2021
Bayes in the sky: Bayesian inference and model selection in cosmology Trotta, Roberto Contemporary Physics, Vol. 49, Issue 2 https://doi.org/10.1080/00107510802066753	journal	March 2008
HEPfit: a code for the combination of indirect and direct constraints on high energy physics models de Blas, J.; Chowdhury, D.; Ciuchini, M. The European Physical Journal C, Vol. 80, Issue 5 https://doi.org/10.1140/epjc/s10052-020-7904-z	journal	May 2020
Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC Chatrchyan, S.; Khachatryan, V.; Sirunyan, A. M. Physics Letters B, Vol. 716, Issue 1 https://doi.org/10.1016/j.physletb.2012.08.021	journal	September 2012
The Selection of Prior Distributions by Formal Rules Kass, Robert E.; Wasserman, Larry Journal of the American Statistical Association, Vol. 91, Issue 435 https://doi.org/10.1080/01621459.1996.10477003	journal	September 1996
Frequentist evaluation of intervals estimated for a binomial parameter and for the ratio of Poisson means Cousins, Robert D.; Hymes, Kathryn E.; Tucker, Jordan Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 612, Issue 2 https://doi.org/10.1016/j.nima.2009.10.156	journal	January 2010
dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences Speagle, Joshua S. Monthly Notices of the Royal Astronomical Society, Vol. 493, Issue 3 https://doi.org/10.1093/mnras/staa278	journal	February 2020
The Jeffreys–Lindley paradox and discovery criteria in high energy physics Cousins, Robert D. Synthese, Vol. 194, Issue 2 https://doi.org/10.1007/s11229-014-0525-z	journal	July 2014
Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC Aad, G.; Abajyan, T.; Abbott, B. Physics Letters B, Vol. 716, Issue 1 https://doi.org/10.1016/j.physletb.2012.08.020	journal	September 2012
A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications Balázs, Csaba; van Beekveld, Melissa; Caron, Sascha Journal of High Energy Physics, Vol. 2021, Issue 5 https://doi.org/10.1007/JHEP05(2021)108	journal	May 2021
Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence Berger, James O.; Sellke, Thomas Journal of the American Statistical Association, Vol. 82, Issue 397 https://doi.org/10.2307/2289131	journal	March 1987
An Automatic Method for Finding the Greatest or Least Value of a Function Rosenbrock, H. H. The Computer Journal, Vol. 3, Issue 3 https://doi.org/10.1093/comjnl/3.3.175	journal	March 1960
`emcee` : The MCMC Hammer Foreman-Mackey, Daniel; Hogg, David W.; Lang, Dustin Publications of the Astronomical Society of the Pacific, Vol. 125, Issue 925 https://doi.org/10.1086/670067	journal	March 2013
Improved limits on dark matter annihilation in the Sun with the 79-string IceCube detector and implications for supersymmetry Aartsen, M. G.; Abraham, K.; Ackermann, M. Journal of Cosmology and Astroparticle Physics, Vol. 2016, Issue 04 https://doi.org/10.1088/1475-7516/2016/04/022	journal	April 2016
Reporting results in High Energy Physics publications: A manifesto Vischia, Pietro Reviews in Physics, Vol. 5 https://doi.org/10.1016/j.revip.2020.100046	journal	November 2020
Dark matter direct-detection experiments Undagoitia, Teresa Marrodán; Rauch, Ludwig Journal of Physics G: Nuclear and Particle Physics, Vol. 43, Issue 1 https://doi.org/10.1088/0954-3899/43/1/013001	journal	December 2015
Minuit - a system for function minimization and analysis of the parameter errors and correlations James, F.; Roos, M. Computer Physics Communications, Vol. 10, Issue 6 https://doi.org/10.1016/0010-4655(75)90039-9	journal	December 1975
Abandon Statistical Significance McShane, Blakeley B.; Gal, David; Gelman, Andrew The American Statistician, Vol. 73, Issue sup1 https://doi.org/10.1080/00031305.2018.1527253	journal	March 2019
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations Greenland, Sander; Senn, Stephen J.; Rothman, Kenneth J. European Journal of Epidemiology, Vol. 31, Issue 4 https://doi.org/10.1007/s10654-016-0149-3	journal	April 2016
Justify your alpha Lakens, Daniel; Adolfi, Federico G.; Albers, Casper J. Nature Human Behaviour, Vol. 2, Issue 3 https://doi.org/10.1038/s41562-018-0311-x	journal	February 2018
Efficiently Simulating the Coverage Properties of Interval Estimates Rubin, D. B.; Schenker, N. Applied Statistics, Vol. 35, Issue 2 https://doi.org/10.2307/2347266	journal	January 1986
Use of event-level neutrino telescope data in global fits for theories of new physics Scott, P.; Savage, C.; Edsjö, J. Journal of Cosmology and Astroparticle Physics, Vol. 2012, Issue 11 https://doi.org/10.1088/1475-7516/2012/11/057	journal	November 2012
GAMBIT: the global and modular beyond-the-standard-model inference tool Athron, Peter; Balazs, Csaba; Bringmann, Torsten The European Physical Journal C, Vol. 77, Issue 11 https://doi.org/10.1140/epjc/s10052-017-5321-8	journal	November 2017
Genetic Algorithms in Astronomy and Astrophysics Charbonneau, P. The Astrophysical Journal Supplement Series, Vol. 101 https://doi.org/10.1086/192242	journal	November 1995
Natural priors, CMSSM fits and LHC weather forecasts Allanach, Benjamin C.; Cranmer, Kyle; Lester, Christopher G. Journal of High Energy Physics, Vol. 2007, Issue 08 https://doi.org/10.1088/1126-6708/2007/08/023	journal	August 2007
MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics Feroz, F.; Hobson, M. P.; Bridges, M. Monthly Notices of the Royal Astronomical Society, Vol. 398, Issue 4 https://doi.org/10.1111/j.1365-2966.2009.14548.x	journal	October 2009
Reproducibility and Replication of Experimental Particle Physics Results Junk, Thomas R.; Lyons, Louis Issue 2.4, Fall 2020, Vol. 2, Issue 4 https://doi.org/10.1162/99608f92.250f995b	journal	December 2020
Two cheers for P-values? Senn, S. Journal of Epidemiology and Biostatistics, Vol. 6, Issue 2 https://doi.org/10.1080/135952201753172953	journal	March 2001
HEPLike: An open source framework for experimental likelihood evaluation Bhom, Jihyun; Chrzaszcz, Marcin Computer Physics Communications, Vol. 254 https://doi.org/10.1016/j.cpc.2020.107235	journal	September 2020
Searching for new phenomena with profile likelihood ratio tests Algeri, Sara; Aalbers, Jelle; Morå, Knut Dundas Nature Reviews Physics, Vol. 2, Issue 5 https://doi.org/10.1038/s42254-020-0169-5	journal	April 2020
Data Analysis Recipes: Using Markov Chain Monte Carlo Hogg, David W.; Foreman-Mackey, Daniel The Astrophysical Journal Supplement Series, Vol. 236, Issue 1 https://doi.org/10.3847/1538-4365/aab76e	journal	May 2018
The Practical Alternative to the p Value Is the Correctly Used p Value Lakens, Daniël Perspectives on Psychological Science, Vol. 16, Issue 3 https://doi.org/10.1177/1745691620958012	journal	February 2021
2000 CKM-triangle analysis a critical review with updated experimental inputs and theoretical parameters Ciuchini, Marco; Lubicz, Vittorio; D'Agostini, Giulio Journal of High Energy Physics, Vol. 2001, Issue 07 https://doi.org/10.1088/1126-6708/2001/07/013	journal	July 2001
The global electroweak fit at NNLO and prospects for the LHC and ILC Baak, M.; Cúth, J.; Haller, J. The European Physical Journal C, Vol. 74, Issue 9 https://doi.org/10.1140/epjc/s10052-014-3046-5	journal	September 2014
Statistical coverage for supersymmetric parameter estimation: a case study with direct detection of dark matter Akrami, Yashar; Savage, Christopher; Scott, Pat Journal of Cosmology and Astroparticle Physics, Vol. 2011, Issue 07 https://doi.org/10.1088/1475-7516/2011/07/002	journal	July 2011
Review of Particle Physics Zyla, P. A.; Barnett, R. M.; Beringer, J. Progress of Theoretical and Experimental Physics, Vol. 2020, Issue 8 https://doi.org/10.1093/ptep/ptaa104	journal	August 2020
In defense ofPvalues Murtaugh, Paul A. Ecology, Vol. 95, Issue 3 https://doi.org/10.1890/13-0590.1	journal	March 2014
The fallacy of placing confidence in confidence intervals Morey, Richard D.; Hoekstra, Rink; Rouder, Jeffrey N. Psychonomic Bulletin & Review, Vol. 23, Issue 1 https://doi.org/10.3758/s13423-015-0947-8	journal	October 2015
polychord: nested sampling for cosmology Handley, W. J.; Hobson, M. P.; Lasenby, A. N. Monthly Notices of the Royal Astronomical Society: Letters, Vol. 450, Issue 1 https://doi.org/10.1093/mnrasl/slv047	journal	April 2015
Planck 2018 results: V. CMB power spectra and likelihoods Aghanim, N.; Akrami, Y.; Ashdown, M. Astronomy & Astrophysics, Vol. 641 https://doi.org/10.1051/0004-6361/201936386	journal	September 2020
A Markov chain Monte Carlo analysis of the CMSSM Austri, Roberto Ruiz de; Trotta, Roberto; Roszkowski, Leszek Journal of High Energy Physics, Vol. 2006, Issue 05 https://doi.org/10.1088/1126-6708/2006/05/002	journal	May 2006
Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module Martinez, Gregory D.; McKay, James; Farmer, Ben The European Physical Journal C, Vol. 77, Issue 11 https://doi.org/10.1140/epjc/s10052-017-5274-y	journal	November 2017
A Statistical Paradox Lindley, D. V. Biometrika, Vol. 44, Issue 1-2 https://doi.org/10.1093/biomet/44.1-2.187	journal	January 1957
The Weighted Likelihood Ratio, Linear Hypotheses on Normal Location Parameters Dickey, James M. The Annals of Mathematical Statistics, Vol. 42, Issue 1 https://doi.org/10.1214/aoms/1177693507	journal	February 1971
Higgs and supersymmetry Buchmueller, O.; Cavanaugh, R.; De Roeck, A. The European Physical Journal C, Vol. 72, Issue 6 https://doi.org/10.1140/epjc/s10052-012-2020-3	journal	June 2012
Trial factors for the look elsewhere effect in high energy physics Gross, Eilam; Vitells, Ofer The European Physical Journal C, Vol. 70, Issue 1-2 https://doi.org/10.1140/epjc/s10052-010-1470-8	journal	October 2010
Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability Neyman, J. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 236, Issue 767 https://doi.org/10.1098/rsta.1937.0005	journal	August 1937
Author Correction: SciPy 1.0: fundamental algorithms for scientific computing in Python Virtanen, Pauli; Gommers, Ralf; Oliphant, Travis E. Nature Methods, Vol. 17, Issue 3 https://doi.org/10.1038/s41592-020-0772-5	journal	February 2020
A Dirty Dozen: Twelve P-Value Misconceptions Goodman, Steven Seminars in Hematology, Vol. 45, Issue 3 https://doi.org/10.1053/j.seminhematol.2008.04.003	journal	July 2008
Confusion Over Measures of Evidence ( p 's) Versus Errors ( α 's) in Classical Statistical Testing Hubbard, Raymond; Bayarri, M. J. The American Statistician, Vol. 57, Issue 3 https://doi.org/10.1198/0003130031856	journal	August 2003
LikeDM: Likelihood calculator of dark matter detection Huang, Xiaoyuan; Tsai, Yue-Lin Sming; Yuan, Qiang Computer Physics Communications, Vol. 213 https://doi.org/10.1016/j.cpc.2016.12.015	journal	April 2017
Why P Values Are Not a Useful Measure of Evidence in Statistical Significance Testing Hubbard, Raymond; Lindsay, R. Murray Theory & Psychology, Vol. 18, Issue 1 https://doi.org/10.1177/0959354307086923	journal	February 2008
The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses Wilks, S. S. The Annals of Mathematical Statistics, Vol. 9, Issue 1 https://doi.org/10.1214/aoms/1177732360	journal	March 1938
A Simplex Method for Function Minimization Nelder, J. A.; Mead, R. The Computer Journal, Vol. 7, Issue 4 https://doi.org/10.1093/comjnl/7.4.308	journal	January 1965
MontePython 3: Boosted MCMC sampler and other features Brinckmann, Thejs; Lesgourgues, Julien Physics of the Dark Universe, Vol. 24 https://doi.org/10.1016/j.dark.2018.100260	journal	March 2019
Erratum to: Asymptotic formulae for likelihood-based tests of new physics Cowan, Glen; Cranmer, Kyle; Gross, Eilam The European Physical Journal C, Vol. 73, Issue 7 https://doi.org/10.1140/epjc/s10052-013-2501-z	journal	July 2013
Presentation of search results: the CL _s technique Read, A. L. Journal of Physics G: Nuclear and Particle Physics, Vol. 28, Issue 10 https://doi.org/10.1088/0954-3899/28/10/313	journal	September 2002
Asymptotic formulae for likelihood-based tests of new physics Cowan, Glen; Cranmer, Kyle; Gross, Eilam The European Physical Journal C, Vol. 71, Issue 2 https://doi.org/10.1140/epjc/s10052-011-1554-0	journal	February 2011
Optimization by Simulated Annealing Kirkpatrick, S.; Gelatt, C. D.; Vecchi, M. P. Science, Vol. 220, Issue 4598 https://doi.org/10.1126/science.220.4598.671	journal	May 1983
dynesty: a dynamic nested sampling package for estimating Bayesian posteriors and evidences Speagle, Joshua S. Monthly Notices of the Royal Astronomical Society, Vol. 493, Issue 3 https://doi.org/10.1093/mnras/staa278	journal	February 2020
Redefine statistical significance Benjamin, Daniel J.; Berger, James O.; Johannesson, Magnus Nature Human Behaviour, Vol. 2, Issue 1 https://doi.org/10.1038/s41562-017-0189-z	journal	September 2017
ColliderBit: a GAMBIT module for the calculation of high-energy collider observables and likelihoods Balázs, Csaba; Buckley, Andy; Dal, Lars A. The European Physical Journal C, Vol. 77, Issue 11 https://doi.org/10.1140/epjc/s10052-017-5285-8	journal	November 2017
Limits and confidence intervals in the presence of nuisance parameters Rolke, Wolfgang A.; López, Angel M.; Conrad, Jan Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 551, Issue 2-3 https://doi.org/10.1016/j.nima.2005.05.068	journal	October 2005
Comment on “Bayesian Analysis of Pentaquark Signals from CLAS Data” Cousins, Robert D. Physical Review Letters, Vol. 101, Issue 2 https://doi.org/10.1103/physrevlett.101.029101	journal	July 2008