Self-calibrating the look-elsewhere effect: fast evaluation of the statistical significance using peak heights

Bayer, Adrian E.; Seljak, Uroš; Robnik, Jakob

doi:10.1093/mnras/stab2331

Self-calibrating the look-elsewhere effect: fast evaluation of the statistical significance using peak heights

Journal Article · Tue Sep 21 00:00:00 EDT 2021 · Monthly Notices of the Royal Astronomical Society

DOI:https://doi.org/10.1093/mnras/stab2331· OSTI ID:1901544

^[1]; Seljak, Uroš ^[2]; Robnik, Jakob ^[3]

Univ. of California, Berkeley, CA (United States)
Univ. of California, Berkeley, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Eidgenoessische Technische Hochschule (ETH), Zurich (Switzerland)

In experiments where one searches a large parameter space for an anomaly, one often finds many spurious noise-induced peaks in the likelihood. This is known as the look-elsewhere effect, and must be corrected for when performing statistical analysis. Here, this paper introduces a method to calibrate the false alarm probability (FAP), or p-value, for a given dataset by considering the heights of the highest peaks in the likelihood. Specifically, we derive an equation relating the global p-value to the rank and height of local maxima. In the simplest form of self-calibration, the look-elsewhere-corrected $$\chi^2$$ of a physical peak is approximated by the $$\chi^2$$ of the peak minus the $$\chi^2$$ of the highest noise-induced peak, with accuracy improved by considering lower peaks. In contrast to alternative methods, this approach has negligible computational cost as peaks in the likelihood are a byproduct of every peak-search analysis. We apply to examples from astronomy, including planet detection, periodograms, and cosmology.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: National Aeronautics and Space Administration (NASA); National Science Foundation (NSF); USDOE Office of Science (SC), High Energy Physics (HEP)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1901544

Journal Information:: Monthly Notices of the Royal Astronomical Society, Journal Name: Monthly Notices of the Royal Astronomical Society Journal Issue: 1 Vol. 508; ISSN 0035-8711

Publisher:: Royal Astronomical SocietyCopyright Statement

Country of Publication:: United States

Language:: English

References (48)

Mathematical Analysis of Random Noise Rice, S. O. Bell System Technical Journal, Vol. 24, Issue 1 https://doi.org/10.1002/j.1538-7305.1945.tb00453.x	journal	January 1945
Least-squares frequency analysis of unequally spaced data Lomb, N. R. Astrophysics and Space Science, Vol. 39, Issue 2 https://doi.org/10.1007/BF00648343	journal	February 1976
Fine-resolution analysis of exoplanetary distributions by wavelets: hints of an overshooting iceline accumulation Baluev, Roman V.; Shaidulin, Vakhit S. Astrophysics and Space Science, Vol. 363, Issue 9 https://doi.org/10.1007/s10509-018-3416-9	journal	August 2018
Analysing the Main Belt asteroid distributions by wavelets Baluev, R. V.; Rodionov, E. I. Celestial Mechanics and Dynamical Astronomy, Vol. 132, Issue 6-7 https://doi.org/10.1007/s10569-020-09976-2	journal	June 2020
Practical Guidelines for Multiplicity Adjustment in Clinical Trials Proschan, Michael A.; Waclawiw, Myron A. Controlled Clinical Trials, Vol. 21, Issue 6 https://doi.org/10.1016/S0197-2456(00)00106-9	journal	December 2000
Statistical detection of patterns in unidimensional distributions by continuous wavelet transforms Baluev, R. V. Astronomy and Computing, Vol. 23 https://doi.org/10.1016/j.ascom.2018.04.004	journal	April 2018
Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC Aad, G.; Abajyan, T.; Abbott, B. Physics Letters B, Vol. 716, Issue 1 https://doi.org/10.1016/j.physletb.2012.08.020	journal	September 2012
Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC Chatrchyan, S.; Khachatryan, V.; Sirunyan, A. M. Physics Letters B, Vol. 716, Issue 1 https://doi.org/10.1016/j.physletb.2012.08.021	journal	September 2012
Efficiency and power in genetic association studies de Bakker, Paul I. W.; Yelensky, Roman; Pe'er, Itsik Nature Genetics, Vol. 37, Issue 11 https://doi.org/10.1038/ng1669	journal	October 2005
Planck 2013 results. I. Overview of products and scientific results Ade, P. A. R.; Aghanim, N.; Alves, M. I. R. Astronomy & Astrophysics, Vol. 571 https://doi.org/10.1051/0004-6361/201321529	journal	October 2014
Planck 2013 results. XV. CMB power spectra and likelihood Ade, P. A. R.; Aghanim, N.; Armitage-Caplan, C. Astronomy & Astrophysics, Vol. 571 https://doi.org/10.1051/0004-6361/201321573	journal	October 2014
Efficient modeling of correlated noise Delisle, J. -B.; Hara, N.; Ségransan, D. Astronomy & Astrophysics, Vol. 635 https://doi.org/10.1051/0004-6361/201936905	journal	March 2020
Statistical significance for genomewide studies Storey, J. D.; Tibshirani, R. Proceedings of the National Academy of Sciences, Vol. 100, Issue 16, p. 9440-9445 https://doi.org/10.1073/pnas.1530509100	journal	July 2003
Rectangular Confidence Regions for the Means of Multivariate Normal Distributions Šidák, Zbyněk Journal of the American Statistical Association, Vol. 62, Issue 318 https://doi.org/10.1080/01621459.1967.10482935	journal	June 1967
The cleanest Fourier spectrum Foster, Grant The Astronomical Journal, Vol. 109 https://doi.org/10.1086/117416	journal	April 1995
Studies in astronomical time series analysis. II - Statistical aspects of spectral analysis of unevenly spaced data Scargle, J. D. The Astrophysical Journal, Vol. 263 https://doi.org/10.1086/160554	journal	December 1982
Efficient Computation of Cosmic Microwave Background Anisotropies in Closed Friedmann‐Robertson‐Walker Models Lewis, Antony; Challinor, Anthony; Lasenby, Anthony The Astrophysical Journal, Vol. 538, Issue 2 https://doi.org/10.1086/309179	journal	August 2000
Do high energy astrophysical neutrinos trace star formation? Emig, Kimberly; Lunardini, Cecilia; Windhorst, Rogier Journal of Cosmology and Astroparticle Physics, Vol. 2015, Issue 12 https://doi.org/10.1088/1475-7516/2015/12/029	journal	December 2015
Search for features in the spectrum of primordial perturbations using Planck and other datasets Hunt, Paul; Sarkar, Subir Journal of Cosmology and Astroparticle Physics, Vol. 2015, Issue 12 https://doi.org/10.1088/1475-7516/2015/12/052	journal	December 2015
Search for gamma-ray lines towards galaxy clusters with the Fermi-LAT Anderson, B.; Zimmer, S.; Conrad, J. Journal of Cosmology and Astroparticle Physics, Vol. 2016, Issue 02 https://doi.org/10.1088/1475-7516/2016/02/026	journal	February 2016
A precision search for WIMPs with charged cosmic rays Reinert, Annika; Winkler, Martin Wolfgang Journal of Cosmology and Astroparticle Physics, Vol. 2018, Issue 01 https://doi.org/10.1088/1475-7516/2018/01/055	journal	January 2018
The look-elsewhere effect from a unified Bayesian and frequentist perspective Bayer, Adrian E.; Seljak, Uroš Journal of Cosmology and Astroparticle Physics, Vol. 2020, Issue 10 https://doi.org/10.1088/1475-7516/2020/10/009	journal	October 2020
On methods for correcting for the look-elsewhere effect in searches for new physics Algeri, S.; Dyk, D. A. van; Conrad, J. Journal of Instrumentation, Vol. 11, Issue 12 https://doi.org/10.1088/1748-0221/11/12/p12010	journal	December 2016
KEPLER MISSION DESIGN, REALIZED PHOTOMETRIC PERFORMANCE, AND EARLY SCIENCE Koch, David G.; Borucki, William J.; Basri, Gibor The Astrophysical Journal, Vol. 713, Issue 2 https://doi.org/10.1088/2041-8205/713/2/L79	journal	March 2010
Hypothesis testing when a nuisance parameter is present only under the alternative Davies, Robert B. Biometrika, Vol. 74, Issue 1 https://doi.org/10.1093/biomet/74.1.33	journal	January 1987
Matched filtering with non-Gaussian noise for planet transit detections Robnik, Jakob; Seljak, Uroš Monthly Notices of the Royal Astronomical Society, Vol. 504, Issue 4 https://doi.org/10.1093/mnras/stab1178	journal	April 2021
Detecting non-sinusoidal periodicities in observational data: the von Mises periodogram for variable stars and exoplanetary transits Baluev, Roman V. Monthly Notices of the Royal Astronomical Society, Vol. 431, Issue 2 https://doi.org/10.1093/mnras/stt238	journal	March 2013
Keplerian periodogram for Doppler exoplanet detection: optimized computation and analytic significance thresholds Baluev, Roman V. Monthly Notices of the Royal Astronomical Society, Vol. 446, Issue 2 https://doi.org/10.1093/mnras/stu2191	journal	November 2014
Planes of satellite galaxies: when exceptions are the rule Cautun, Marius; Bose, Sownak; Frenk, Carlos S. Monthly Notices of the Royal Astronomical Society, Vol. 452, Issue 4 https://doi.org/10.1093/mnras/stv1557	journal	August 2015
A search for a keV signature of radiatively decaying dark matter with Suzaku XIS observations of the X-ray diffuse background Sekiya, Norio; Yamasaki, Noriko Y.; Mitsuda, Kazuhisa Publications of the Astronomical Society of Japan, Vol. 68, Issue SP1 https://doi.org/10.1093/pasj/psv081	journal	September 2015
Periodogram and likelihood periodicity search in the SNO solar neutrino data Ranucci, Gioacchino; Rovere, Marco Physical Review D, Vol. 75, Issue 1 https://doi.org/10.1103/PhysRevD.75.013010	journal	January 2007
Combining power spectrum and bispectrum measurements to detect oscillatory features Fergusson, J. R.; Gruetjen, H. F.; Shellard, E. P. S. Physical Review D, Vol. 91, Issue 2 https://doi.org/10.1103/PhysRevD.91.023502	journal	January 2015
Polyspectra searches for sharp oscillatory features in cosmic microwave sky data Fergusson, J. R.; Gruetjen, H. F.; Shellard, E. P. S. Physical Review D, Vol. 91, Issue 12 https://doi.org/10.1103/PhysRevD.91.123506	journal	June 2015
Cosmological parameters from CMB and other data: A Monte Carlo approach Lewis, Antony; Bridle, Sarah Physical Review D, Vol. 66, Issue 10 https://doi.org/10.1103/physrevd.66.103511	journal	November 2002
GW150914: First results from the search for binary black hole coalescence with Advanced LIGO Abbott, B. P.; Abbott, R.; Abbott, T. D. Physical Review D, Vol. 93, Issue 12 https://doi.org/10.1103/physrevd.93.122003	journal	June 2016
Analysis framework for the prompt discovery of compact binary mergers in gravitational-wave data Messick, Cody; Blackburn, Kent; Brady, Patrick Physical Review D, Vol. 95, Issue 4 https://doi.org/10.1103/physrevd.95.042001	journal	February 2017
Observation of High-Energy Astrophysical Neutrinos in Three Years of IceCube Data Aartsen, M. G.; Ackermann, M.; Adams, J. Physical Review Letters, Vol. 113, Issue 10 https://doi.org/10.1103/physrevlett.113.101101	journal	September 2014
Assessing the statistical significance of periodogram peaks Baluev, R. V. Monthly Notices of the Royal Astronomical Society, Vol. 385, Issue 3 https://doi.org/10.1111/j.1365-2966.2008.12689.x	journal	April 2008
Detecting non-sinusoidal periodicities in observational data using multiharmonic periodograms Baluev, Roman V. Monthly Notices of the Royal Astronomical Society, Vol. 395, Issue 3 https://doi.org/10.1111/j.1365-2966.2009.14634.x	journal	May 2009
Kepler Planet-Detection Mission: Introduction and First Results Borucki, W. J.; Koch, D.; Basri, G. Science, Vol. 327, Issue 5968 https://doi.org/10.1126/science.1185402	journal	January 2010
Trial factors for the look elsewhere effect in high energy physics Gross, Eilam; Vitells, Ofer The European Physical Journal C, Vol. 70, Issue 1-2 https://doi.org/10.1140/epjc/s10052-010-1470-8	journal	October 2010
Asymptotic formulae for likelihood-based tests of new physics Cowan, Glen; Cranmer, Kyle; Gross, Eilam The European Physical Journal C, Vol. 71, Issue 2 https://doi.org/10.1140/epjc/s10052-011-1554-0	journal	February 2011
Multiple Hypothesis Testing Shaffer, J. P. Annual Review of Psychology, Vol. 46, Issue 1 https://doi.org/10.1146/annurev.ps.46.020195.003021	journal	January 1995
Solving the Bible Code Puzzle Bar-Hillel, Maya; Bar-Natan, Dror; Kalai, Gil Statistical Science, Vol. 14, Issue 2 https://doi.org/10.1214/ss/1009212243	journal	May 1999
Memoir on the Probability of the Causes of Events Laplace, Pierre Simon Statistical Science, Vol. 1, Issue 3 https://doi.org/10.1214/ss/1177013621	journal	August 1986
Hypothesis Testing When a Nuisance Parameter is Present Only Under the Alternative Davies, R. B. Biometrika, Vol. 64, Issue 2 https://doi.org/10.2307/2335690	journal	August 1977
Kepler Data Analysis: Non-Gaussian Noise and Fourier Gaussian Process Analysis of Stellar Variability Robnik, Jakob; Seljak, Uroš The Astronomical Journal, Vol. 159, Issue 5 https://doi.org/10.3847/1538-3881/ab8460	journal	April 2020
Understanding the Lomb–Scargle Periodogram VanderPlas, Jacob T. The Astrophysical Journal Supplement Series, Vol. 236, Issue 1 https://doi.org/10.3847/1538-4365/aab766	journal	May 2018

Similar Records

The look-elsewhere effect from a unified Bayesian and frequentist perspective

Journal Article · Thu Oct 01 20:00:00 EDT 2020 · Journal of Cosmology and Astroparticle Physics · OSTI ID:1838321

Fast estimation of the look-elsewhere effect using Gaussian random fields

Journal Article · Sun Nov 12 19:00:00 EST 2023 · European Physical Journal. C, Particles and Fields (Online) · OSTI ID:2205524

Unaccounted-for look-elsewhere effect in k-fold cross adaptive anomaly searches

Conference · Mon Jul 07 00:00:00 EDT 2025 · No journal information · OSTI ID:2571312

Related Subjects

79 ASTRONOMY AND ASTROPHYSICS
astroparticle physics
data analysis
inflation
planetary systems
planets and satellites
statistics

Self-calibrating the look-elsewhere effect: fast evaluation of the statistical significance using peak heights

Citation Formats

References (48)

Similar Records

Related Subjects