Convergence Rates for Empirical Estimation of Binary Classification Bounds

Sekeh, Salimeh Yasaei; Noshad, Morteza; Moon, Kevin R.; Hero, Alfred O.

doi:10.3390/e21121144

Title: Convergence Rates for Empirical Estimation of Binary Classification Bounds

Journal Article · 22 November 2019 · Entropy

DOI: https://doi.org/10.3390/e21121144 · OSTI ID:1801119

^[1]; Noshad, Morteza ^[2]; Moon, Kevin R. ^[3];

^[2]

Univ. of Maine, Orono, ME (United States). School of Computing and Information Science; OSTI
Univ. of Michigan, Ann Arbor, MI (United States). Dept. of Electrical Engineering and Computer Science
Utah State Univ., Logan, UT (United States). Dept. of Mathematics and Statistics

Bounding the best achievable error probability for binary classification problems is relevant to many applications including machine learning, signal processing, and information theory. Many bounds on the Bayes binary classification error rate depend on information divergences between the pair of class distributions. Recently, the Henze–Penrose (HP) divergence has been proposed for bounding classification error probability. We consider the problem of empirically estimating the HP-divergence from random samples. We derive a bound on the convergence rate for the Friedman–Rafsky (FR) estimator of the HP-divergence, which is related to a multivariate runs statistic for testing between two distributions. The FR estimator is derived from a multicolored Euclidean minimal spanning tree (MST) that spans the merged samples. We obtain a concentration inequality for the Friedman–Rafsky estimator of the Henze–Penrose divergence. We validate our results experimentally and illustrate their application to real datasets.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Georgia Institute of Technology, Atlanta, GA (United States); Univ. of Michigan, Ann Arbor, MI (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: NA0002534; NA0003921

OSTI ID:: 1801119

Journal Information:: Entropy, Journal Name: Entropy Journal Issue: 12 Vol. 21; ISSN ENTRFG; ISSN 1099-4300

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

References (34)

Asymptotics for Euclidean functionals with power-weighted edges Redmond, C.; Yukich, J. E. Stochastic Processes and their Applications, Vol. 61, Issue 2 https://doi.org/10.1016/0304-4149(95)00075-5	journal	February 1996
Image matching using alpha-entropy measures and entropic graphs Neemuchwala, Huzefa; Hero, Alfred; Carson, Paul Signal Processing, Vol. 85, Issue 2 https://doi.org/10.1016/j.sigpro.2004.10.002	journal	February 2005
Divergence measures for statistical data processing—An annotated bibliography Basseville, Michèle Signal Processing, Vol. 93, Issue 4 https://doi.org/10.1016/j.sigpro.2012.09.003	journal	April 2013
On the number of leaves of a euclidean minimal spanning tree Steele, J. Michael; Shepp, Lawrence A.; Eddy, William F. Journal of Applied Probability, Vol. 24, Issue 04 https://doi.org/10.1017/s0021900200116705	journal	December 1987
A brief survey of time- and frequency-domain adaptive filters Spanias, Andreas 2016 7th International Conference on Information, Intelligence, Systems & Applications (IISA) https://doi.org/10.1109/iisa.2016.7785389	conference	July 2016
Exponential Concentration of a Density Functional Estimator Singh, Shashank; Poczos, Barnabas Carnegie Mellon University https://doi.org/10.1184/r1/6475619	text	January 2014
Generalized Exponential Concentration Inequality for Renyi Divergence Estimation Singh, Shashank; Poczos, Barnabas Carnegie Mellon University https://doi.org/10.1184/r1/6475736	text	January 2014
Limit Theorems and Rates of Convergence for Euclidean Functionals Redmond, C.; Yukich, J. E. The Annals of Applied Probability, Vol. 4, Issue 4 https://doi.org/10.1214/aoap/1177004902	journal	November 1994
On a Test Whether Two Samples are from the Same Population Wald, A.; Wolfowitz, J. The Annals of Mathematical Statistics, Vol. 11, Issue 2 https://doi.org/10.1214/aoms/1177731909	journal	June 1940
Employees’ Motivation and Management of Human Resources in Public Enterprises in Dukagjin’s Plain Berisha, Ardian European Journal of Economics and Business Studies, Vol. 6, Issue 1 https://doi.org/10.26417/ejes.v6i1.p169-174	journal	December 2016
Asymptotics for Euclidean minimal spanning trees on random points Aldous, David; Steele, J. Michael Probability Theory and Related Fields, Vol. 92, Issue 2 https://doi.org/10.1007/BF01194923	journal	June 1992
Concentration of measure and isoperimetric inequalities in product spaces Talagrand, Michel Publications mathématiques de l'IHÉS, Vol. 81, Issue 1 https://doi.org/10.1007/BF02699376	journal	December 1995
The relative neighbourhood graph of a finite planar set Toussaint, Godfried T. Pattern Recognition, Vol. 12, Issue 4 https://doi.org/10.1016/0031-3203(80)90066-7	journal	January 1980
A test of randomness based on the minimal spanning tree Hoffman, Richard; Jain, Anil K. Pattern Recognition Letters, Vol. 1, Issue 3 https://doi.org/10.1016/0167-8655(83)90059-4	journal	March 1983
Asymptotics for Euclidean functionals with power-weighted edges Redmond, C.; Yukich, J. E. Stochastic Processes and their Applications, Vol. 61, Issue 2 https://doi.org/10.1016/0304-4149(95)00075-5	journal	February 1996
Image matching using alpha-entropy measures and entropic graphs Neemuchwala, Huzefa; Hero, Alfred; Carson, Paul Signal Processing, Vol. 85, Issue 2 https://doi.org/10.1016/j.sigpro.2004.10.002	journal	February 2005
Divergence measures for statistical data processing—An annotated bibliography Basseville, Michèle Signal Processing, Vol. 93, Issue 4 https://doi.org/10.1016/j.sigpro.2012.09.003	journal	April 2013
On the number of leaves of a euclidean minimal spanning tree Steele, J. Michael; Shepp, Lawrence A.; Eddy, William F. Journal of Applied Probability, Vol. 24, Issue 04 https://doi.org/10.1017/s0021900200116705	journal	December 1987
A synthetic energy dataset for non-intrusive load monitoring in households Klemenjak, Christoph; Kovatsch, Christoph; Herold, Manuel Scientific Data, Vol. 7, Issue 1 https://doi.org/10.1038/s41597-020-0434-6	journal	April 2020
Divergence measures based on the Shannon entropy Lin, J. IEEE Transactions on Information Theory, Vol. 37, Issue 1 https://doi.org/10.1109/18.61115	journal	January 1991
Blind source separation using Renyi's mutual information Hild, K. E.; Erdogmus, D.; Principe, J. IEEE Signal Processing Letters, Vol. 8, Issue 6 https://doi.org/10.1109/97.923043	journal	June 2001
Empirical Non-Parametric Estimation of the Fisher Information Berisha, Visar; Hero, Alfred O. IEEE Signal Processing Letters, Vol. 22, Issue 7 https://doi.org/10.1109/LSP.2014.2378514	journal	July 2015
Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters Zahn, C. T. IEEE Transactions on Computers, Vol. C-20, Issue 1 https://doi.org/10.1109/T-C.1971.223083	journal	January 1971
Empirically Estimable Classification Bounds Based on a Nonparametric Divergence Measure Berisha, Visar; Wisler, Alan; Hero, Alfred O. IEEE Transactions on Signal Processing, Vol. 64, Issue 3 https://doi.org/10.1109/TSP.2015.2477805	journal	February 2016
A General Class of Coefficients of Divergence of One Distribution from Another Ali, S. M.; Silvey, S. D. Journal of the Royal Statistical Society: Series B (Methodological), Vol. 28, Issue 1 https://doi.org/10.1111/j.2517-6161.1966.tb00626.x	journal	January 1966
Limit Theorems and Rates of Convergence for Euclidean Functionals Redmond, C.; Yukich, J. E. The Annals of Applied Probability, Vol. 4, Issue 4 https://doi.org/10.1214/aoap/1177004902	journal	November 1994
A Matching Problem and Subadditive Euclidean Functionals Rhee, WanSoo T. The Annals of Applied Probability, Vol. 3, Issue 3 https://doi.org/10.1214/aoap/1177005364	journal	August 1993
On Information and Sufficiency Kullback, S.; Leibler, R. A. The Annals of Mathematical Statistics, Vol. 22, Issue 1 https://doi.org/10.1214/aoms/1177729694	journal	March 1951
On a Test Whether Two Samples are from the Same Population Wald, A.; Wolfowitz, J. The Annals of Mathematical Statistics, Vol. 11, Issue 2 https://doi.org/10.1214/aoms/1177731909	journal	June 1940
On the multivariate runs test Penrose, Mathew D.; Henze, Norbert The Annals of Statistics, Vol. 27, Issue 1 https://doi.org/10.1214/aos/1018031112	journal	March 1999
The Jackknife Estimate of Variance Efron, B.; Stein, C. The Annals of Statistics, Vol. 9, Issue 3 https://doi.org/10.1214/aos/1176345462	journal	May 1981
An Efron-Stein Inequality for Nonsymmetric Statistics Steele, J. Michael The Annals of Statistics, Vol. 14, Issue 2 https://doi.org/10.1214/aos/1176349952	journal	June 1986
On the number of leaves of a euclidean minimal spanning tree Steele, J. Michael; Shepp, Lawrence A.; Eddy, William F. Journal of Applied Probability, Vol. 24, Issue 4 https://doi.org/10.2307/3214207	journal	December 1987
Concentration of Measure and Isoperimetric Inequalities in Product Spaces Talagrand, Michel arXiv https://doi.org/10.48550/arxiv.math/9406212	preprint	January 1994

Similar Records

Geometric Estimation of Multivariate Dependency

Journal Article · 2019 · Entropy · OSTI ID:1614538

Yasaei Sekeh, Salimeh; Hero, Alfred O.

Automated infrared detection of organophosphorus compounds in multicomponent solutions. Master`s thesis

Thesis/Dissertation · 1995 · OSTI ID:251180

Schuchardt, D A

On the time growth of the error of the DG method for advective problems

Journal Article · 2018 · IMA Journal of Numerical Analysis · OSTI ID:1609968

Kučera, Václav; Shu, Chi-Wang

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayes error rate
Friedman–Rafsky test statistic
Henze–Penrose divergence
Physics
bias and variance trade-off
classification
concentration bounds
convergence rates
minimal spanning trees

Title: Convergence Rates for Empirical Estimation of Binary Classification Bounds

Citation Formats

References (34)

Similar Records

Related Subjects