Generalizing Information to the Evolution of Rational Belief

Duersch, Jed A.; Catanach, Thomas A.

doi:10.3390/e22010108

Generalizing Information to the Evolution of Rational Belief

Journal Article · Wed Jan 15 23:00:00 EST 2020 · Entropy

DOI:https://doi.org/10.3390/e22010108· OSTI ID:1605735

^[1]; Catanach, Thomas A. ^[1]

Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome’s plausibility. Information measures based on Shannon’s concept of entropy include realization information, Kullback–Leibler divergence, Lindley’s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-CA), Livermore, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC04-94AL85000

OSTI ID:: 1605735

Alternate ID(s):: OSTI ID: 1618088

Report Number(s):: SAND--2020-3103J; SAND--2019-13298J; 684733

Journal Information:: Entropy, Journal Name: Entropy Journal Issue: 1 Vol. 22; ISSN ENTRFG; ISSN 1099-4300

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

References (39)

Principal Information Theoretic Approaches Soofi, Ehsan S. Journal of the American Statistical Association, Vol. 95, Issue 452 https://doi.org/10.1080/01621459.2000.10474346	journal	December 2000
Information Properties of Order Statistics and Spacings Ebrahimi, N.; Soofi, E. S.; Zahedi, H. IEEE Transactions on Information Theory, Vol. 50, Issue 1 https://doi.org/10.1109/tit.2003.821973	journal	January 2004
On a Measure of the Information Provided by an Experiment Lindley, D. V. The Annals of Mathematical Statistics, Vol. 27, Issue 4 https://doi.org/10.1214/aoms/1177728069	journal	December 1956
On the Distribution Function of Additive Functions Erdos, P. The Annals of Mathematics, Vol. 47, Issue 1 https://doi.org/10.2307/1969031	journal	January 1946
Sur une généralisation des intégrales de M. J. Radon Nikodym, Otton Fundamenta Mathematicae, Vol. 15 https://doi.org/10.4064/fm-15-1-131-179	journal	January 1930
A Mathematical Theory of Communication Shannon, C. E. Bell System Technical Journal, Vol. 27, Issue 4 https://doi.org/10.1002/j.1538-7305.1948.tb00917.x	journal	October 1948
A Mathematical Theory of Communication Shannon, C. E. Bell System Technical Journal, Vol. 27, Issue 3 https://doi.org/10.1002/j.1538-7305.1948.tb01338.x	journal	July 1948
On rational betting systems Adams, Ernest W. Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 1-2 https://doi.org/10.1007/BF02025803	journal	March 1962
On rational betting systems Adams, Ernest W. Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 3-4 https://doi.org/10.1007/bf01969549	journal	April 1964
Understanding predictive information criteria for Bayesian models Gelman, Andrew; Hwang, Jessica; Vehtari, Aki Statistics and Computing, Vol. 24, Issue 6 https://doi.org/10.1007/s11222-013-9416-2	journal	August 2013
Generalized statistics: yet another generalization Jizba, Petr; Arimitsu, Toshihico Physica A: Statistical Mechanics and its Applications, Vol. 340, Issue 1-3 https://doi.org/10.1016/j.physa.2004.03.085	journal	September 2004
Generalized Shannon–Khinchin axioms and uniqueness theorem for pseudo-additive entropies Ilić, Velimir M.; Stanković, Miomir S. Physica A: Statistical Mechanics and its Applications, Vol. 411 https://doi.org/10.1016/j.physa.2014.05.009	journal	October 2014
Friendship stability in adolescence is associated with ventral striatum responses to vicarious rewards Schreuders, Elisabeth; Braams, Barbara R.; Crone, Eveline A. Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-020-20042-1	journal	January 2021
Skillful statistical models to predict seasonal wind speed and solar radiation in a Yangtze River estuary case study Zeng, Peng; Sun, Xun; Farnham, David J. Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-65281-w	journal	May 2020
Kullback-Leibler information as a basis for strong inference in ecological studies Burnham, Kenneth P.; Anderson, David R. Wildlife Research, Vol. 28, Issue 2 https://doi.org/10.1071/WR99107	journal	January 2001
Capturing the Intangible Concept of Information Soofi, Ehsan S. Journal of the American Statistical Association, Vol. 89, Issue 428 https://doi.org/10.1080/01621459.1994.10476865	journal	December 1994
Dynamic Coherence and Probability Kinematics Skyrms, Brian Philosophy of Science, Vol. 54, Issue 1 https://doi.org/10.1086/289350	journal	March 1987
Information Theory and Statistical Mechanics Jaynes, E. T. Physical Review, Vol. 106, Issue 4 https://doi.org/10.1103/PhysRev.106.620	journal	May 1957
Group entropies, correlation laws, and zeta functions Tempesta, Piergiulio Physical Review E, Vol. 84, Issue 2 https://doi.org/10.1103/PhysRevE.84.021121	journal	August 2011
Maximum Entropy Principle in Statistical Inference: Case for Non-Shannonian Entropies Jizba, Petr; Korbel, Jan Physical Review Letters, Vol. 122, Issue 12 https://doi.org/10.1103/PhysRevLett.122.120601	journal	March 2019
Gradient-based learning applied to document recognition Lecun, Y.; Bottou, L.; Bengio, Y. Proceedings of the IEEE, Vol. 86, Issue 11 https://doi.org/10.1109/5.726791	journal	January 1998
A new look at the statistical model identification Akaike, H. IEEE Transactions on Automatic Control, Vol. 19, Issue 6 https://doi.org/10.1109/TAC.1974.1100705	journal	December 1974
Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy Shore, J.; Johnson, R. IEEE Transactions on Information Theory, Vol. 26, Issue 1 https://doi.org/10.1109/TIT.1980.1056144	journal	January 1980
Properties of cross-entropy minimization Shore, J.; Johnson, R. IEEE Transactions on Information Theory, Vol. 27, Issue 4 https://doi.org/10.1109/TIT.1981.1056373	journal	July 1981
Information Properties of Order Statistics and Spacings Ebrahimi, N.; Soofi, E. S.; Zahedi, H. IEEE Transactions on Information Theory, Vol. 50, Issue 1 https://doi.org/10.1109/TIT.2003.821973	journal	January 2004
Information Measures in Perspective: Information Measures in Perspective Ebrahimi, Nader; Soofi, Ehsan S.; Soyer, Refik International Statistical Review, Vol. 78, Issue 3 https://doi.org/10.1111/j.1751-5823.2010.00105.x	journal	December 2010
The Theory of Information Barnard, G. A. Journal of the Royal Statistical Society: Series B (Methodological), Vol. 13, Issue 1 https://doi.org/10.1111/j.2517-6161.1951.tb00069.x	journal	January 1951
Probability, Frequency and Reasonable Expectation Cox, R. T. American Journal of Physics, Vol. 14, Issue 1 https://doi.org/10.1119/1.1990764	journal	January 1946
A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions Hanel, R.; Thurner, S. EPL (Europhysics Letters), Vol. 93, Issue 2 https://doi.org/10.1209/0295-5075/93/20006	journal	January 2011
Bayes' Method for Bookies Freedman, David A.; Purves, Roger A. The Annals of Mathematical Statistics, Vol. 40, Issue 4 https://doi.org/10.1214/aoms/1177697494	journal	August 1969
Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables Good, I. J. The Annals of Mathematical Statistics, Vol. 34, Issue 3 https://doi.org/10.1214/aoms/1177704014	journal	September 1963
On Information and Sufficiency Kullback, S.; Leibler, R. A. The Annals of Mathematical Statistics, Vol. 22, Issue 1 https://doi.org/10.1214/aoms/1177729694	journal	March 1951
Estimating the Dimension of a Model Schwarz, Gideon The Annals of Statistics, Vol. 6, Issue 2 https://doi.org/10.1214/aos/1176344136	journal	March 1978
Expected Information as Expected Utility Bernardo, Jose M. The Annals of Statistics, Vol. 7, Issue 3 https://doi.org/10.1214/aos/1176344689	journal	May 1979
On confirmation and rational betting Lehman, R. Sherman Journal of Symbolic Logic, Vol. 20, Issue 3 https://doi.org/10.2307/2268221	journal	September 1955
Understanding predictive information criteria for Bayesian models Gelman, Andrew; Hwang, Jessica; Vehtari, Aki arXiv https://doi.org/10.48550/arxiv.1307.5928	preprint	January 2013
Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies Ilic, Velimir M.; Stankovic, Miomir S. arXiv https://doi.org/10.48550/arxiv.1311.0323	text	January 2013
Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies Jizba, Petr; Korbel, Jan arXiv https://doi.org/10.48550/arxiv.1808.01172	text	January 2018
Generalized statistics: yet another generalization Jizba, Petr; Arimitsu, Toshihico arXiv https://doi.org/10.48550/arxiv.cond-mat/0312034	text	January 2003

Cited By (1)

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo Catanach, Thomas A.; Vo, Huy D.; Munsky, Brian arXiv https://doi.org/10.48550/arxiv.2001.01373	preprint	January 2020

Similar Records

Jensen-Shannon divergence as a measure of distinguishability between mixed quantum states

Journal Article · Mon Nov 14 23:00:00 EST 2005 · Physical Review. A · OSTI ID:20786462

On variational definition of quantum entropy

Journal Article · Mon Jan 12 23:00:00 EST 2015 · AIP Conference Proceedings · OSTI ID:22390863

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian inference
Kullback–Leibler divergence
Lindley information
entropy
information
maximal uncertainty
mutual information
proper utility
self information

Generalizing Information to the Evolution of Rational Belief

Citation Formats

References (39)

Cited By (1)

Similar Records

Related Subjects