Quantifying Epistemic Uncertainty in Binary Classification via Accuracy Gain

Qian, Christopher; Ganter, Tyler; Michalenko, Joshua; Liang, Feng; Adams, Jason

doi:10.1002/sam.11709

Title: Quantifying Epistemic Uncertainty in Binary Classification via Accuracy Gain

Journal Article · 18 September 2024 · Statistical Analysis and Data Mining

DOI: https://doi.org/10.1002/sam.11709 · OSTI ID:2447035

^[1]; Ganter, Tyler ^[2]; Michalenko, Joshua ^[3];

^[1];

^[4]

Department of Statistics University of Illinois Urbana‐Champaign Champaign Illinois USA
Department of Applied Machine Intelligence Sandia National Laboratories Albuquerque New Mexico USA
Department of Proliferation Signature and Data Exploitation Sandia National Laboratories Albuquerque New Mexico USA
Department of Statistical Sciences Sandia National Laboratories Albuquerque New Mexico USA

ABSTRACT Recently, a surge of interest has been given to quantifying epistemic uncertainty (EU), the reducible portion of uncertainty due to lack of data. We propose a novel EU estimator in the binary classification setting, as the posterior expected value of the empirical gain in accuracy between the current prediction and the optimal prediction. In order to validate the performance of our EU estimator, we introduce an experimental procedure where we take an existing dataset, remove a set of points, and compare the estimated EU with the observed change in accuracy. Through real and simulated data experiments, we demonstrate the effectiveness of our proposed EU estimator.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2447035

Journal Information:: Statistical Analysis and Data Mining, Journal Name: Statistical Analysis and Data Mining Journal Issue: 5 Vol. 17; ISSN 1932-1864

Publisher:: Wiley Blackwell (John Wiley & Sons)Copyright Statement

Country of Publication:: United States

Language:: English

References (11)

Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods Hüllermeier, Eyke; Waegeman, Willem Machine Learning, Vol. 110, Issue 3 https://doi.org/10.1007/s10994-021-05946-3	journal	March 2021
Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty Senge, Robin; Bösner, Stefan; Dembczyński, Krzysztof Information Sciences, Vol. 255 https://doi.org/10.1016/j.ins.2013.07.030	journal	January 2014
Epistemic uncertainty quantification in deep learning classification by the Delta method Nilsen, Geir K.; Munthe-Kaas, Antonella Z.; Skaug, Hans J. Neural Networks, Vol. 145 https://doi.org/10.1016/j.neunet.2021.10.014	journal	January 2022
Searching for exotic particles in high-energy physics with deep learning Baldi, P.; Sadowski, P.; Whiteson, D. Nature Communications, Vol. 5, Issue 1 https://doi.org/10.1038/ncomms5308	journal	July 2014
General Multifidelity Surrogate Models: Framework and Active-Learning Strategies for Efficient Rare Event Simulation Chakroborty, Promit; Dhulipala, Somayajulu L. N.; Che, Yifeng Journal of Engineering Mechanics, Vol. 149, Issue 12 https://doi.org/10.1061/JENMDT.EMENG-7111	journal	December 2023
FINETUNA: fine-tuning accelerated molecular simulations Musielewicz, Joseph; Wang, Xiaoxiao; Tian, Tian Machine Learning: Science and Technology, Vol. 3, Issue 3 https://doi.org/10.1088/2632-2153/ac8fe0	journal	September 2022
The Power of Ensembles for Active Learning in Image Classification Beluch, William H.; Genewein, Tim; Nurnberger, Andreas 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00976	conference	June 2018
Data-Driven Event Detection of Power Systems Based on Unequal-Interval Reduction of PMU Data and Local Outlier Factor Liu, Shengyuan; Zhao, Yuxuan; Lin, Zhenzhi IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2941565	journal	March 2020
On the Statistical Formalism of Uncertainty Quantification Berger, James O.; Smith, Leonard A. Annual Review of Statistics and Its Application, Vol. 6, Issue 1 https://doi.org/10.1146/annurev-statistics-030718-105232	journal	March 2019
Obtaining Well Calibrated Probabilities Using Bayesian Binning Pakdaman Naeini, Mahdi; Cooper, Gregory; Hauskrecht, Milos Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29, Issue 1 https://doi.org/10.1609/aaai.v29i1.9602	journal	February 2015
Reliable Multi-class Classification based on Pairwise Epistemic and Aleatoric Uncertainty Nguyen, Vu-Linh; Destercke, Sébastien; Masson, Marie-Hélène Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2018/706	conference	July 2018

Similar Records

A flexible numerical approach for quantification of epistemic uncertainty

Journal Article · 2013 · Journal of Computational Physics · OSTI ID:22233571

Chen, Xiaoxiao; Park, Eun-Jae; Xiu, Dongbin

Quantifying uncertainty in machine learning for nuclear binding energy

Journal Article · 2025 · Physical Review. C · OSTI ID:3000355

Huang, Mengyao; Wendt, Kyle A.; Schunck, Nicolas F.; +1 more

Composite Multilinearity, Epistemic Uncertainty and Risk Achievement Worth

Journal Article · 2012 · European Journal of Operational Research · OSTI ID:1060424

Borgonovo, E; Smith, C L

Title: Quantifying Epistemic Uncertainty in Binary Classification via Accuracy Gain

Citation Formats

References (11)

Similar Records

Related Subjects