Generalizing Information to the Evolution of Rational Belief

Duersch, Jed A.; Catanach, Thomas A.

doi:10.3390/e22010108

Title: Generalizing Information to the Evolution of Rational Belief

Journal Article · Thu Jan 16 00:00:00 EST 2020 · Entropy

DOI:https://doi.org/10.3390/e22010108· OSTI ID:1605735

^[1]; Catanach, Thomas A. ^[1]

Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome’s plausibility. Information measures based on Shannon’s concept of entropy include realization information, Kullback–Leibler divergence, Lindley’s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC04-94AL85000

OSTI ID:: 1605735

Alternate ID(s):: OSTI ID: 1618088

Report Number(s):: SAND-2020-3103J; SAND-2019-13298J; ENTRFG; 684733

Journal Information:: Entropy, Vol. 22, Issue 1; ISSN 1099-4300

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 4 works

Citation information provided by
Web of Science

References (37)

A Mathematical Theory of Communication Shannon, C. E. Bell System Technical Journal, Vol. 27, Issue 3 https://doi.org/10.1002/j.1538-7305.1948.tb01338.x	journal	July 1948
Estimating the Dimension of a Model Schwarz, Gideon The Annals of Statistics, Vol. 6, Issue 2 https://doi.org/10.1214/aos/1176344136	journal	March 1978
The Theory of Information Barnard, G. A. Journal of the Royal Statistical Society: Series B (Methodological), Vol. 13, Issue 1 https://doi.org/10.1111/j.2517-6161.1951.tb00069.x	journal	January 1951
On rational betting systems Adams, Ernest W. Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 1-2 https://doi.org/10.1007/BF02025803	journal	March 1962
A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions Hanel, R.; Thurner, S. EPL (Europhysics Letters), Vol. 93, Issue 2 https://doi.org/10.1209/0295-5075/93/20006	journal	January 2011
On confirmation and rational betting Lehman, R. Sherman Journal of Symbolic Logic, Vol. 20, Issue 3 https://doi.org/10.2307/2268221	journal	September 1955
On the Distribution Function of Additive Functions Erdos, P. The Annals of Mathematics, Vol. 47, Issue 1 https://doi.org/10.2307/1969031	journal	January 1946
Kullback-Leibler information as a basis for strong inference in ecological studies Burnham, Kenneth P.; Anderson, David R. Wildlife Research, Vol. 28, Issue 2 https://doi.org/10.1071/WR99107	journal	January 2001
Information Theory and Statistical Mechanics Jaynes, E. T. Physical Review, Vol. 106, Issue 4 https://doi.org/10.1103/PhysRev.106.620	journal	May 1957
Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables Good, I. J. The Annals of Mathematical Statistics, Vol. 34, Issue 3 https://doi.org/10.1214/aoms/1177704014	journal	September 1963
Group entropies, correlation laws, and zeta functions Tempesta, Piergiulio Physical Review E, Vol. 84, Issue 2 https://doi.org/10.1103/PhysRevE.84.021121	journal	August 2011
A new look at the statistical model identification Akaike, H. IEEE Transactions on Automatic Control, Vol. 19, Issue 6 https://doi.org/10.1109/TAC.1974.1100705	journal	December 1974
Generalized Shannon–Khinchin axioms and uniqueness theorem for pseudo-additive entropies Ilić, Velimir M.; Stanković, Miomir S. Physica A: Statistical Mechanics and its Applications, Vol. 411 https://doi.org/10.1016/j.physa.2014.05.009	journal	October 2014
On Information and Sufficiency Kullback, S.; Leibler, R. A. The Annals of Mathematical Statistics, Vol. 22, Issue 1 https://doi.org/10.1214/aoms/1177729694	journal	March 1951
Properties of cross-entropy minimization Shore, J.; Johnson, R. IEEE Transactions on Information Theory, Vol. 27, Issue 4 https://doi.org/10.1109/TIT.1981.1056373	journal	July 1981
On a Measure of the Information Provided by an Experiment Lindley, D. V. The Annals of Mathematical Statistics, Vol. 27, Issue 4 https://doi.org/10.1214/aoms/1177728069	journal	December 1956
Understanding predictive information criteria for Bayesian models Gelman, Andrew; Hwang, Jessica; Vehtari, Aki Statistics and Computing, Vol. 24, Issue 6 https://doi.org/10.1007/s11222-013-9416-2	journal	August 2013
Probability, Frequency and Reasonable Expectation Cox, R. T. American Journal of Physics, Vol. 14, Issue 1 https://doi.org/10.1119/1.1990764	journal	January 1946
Principal Information Theoretic Approaches Soofi, Ehsan S. Journal of the American Statistical Association, Vol. 95, Issue 452 https://doi.org/10.1080/01621459.2000.10474346	journal	December 2000
Generalized statistics: yet another generalization Jizba, Petr; Arimitsu, Toshihico Physica A: Statistical Mechanics and its Applications, Vol. 340, Issue 1-3 https://doi.org/10.1016/j.physa.2004.03.085	journal	September 2004
Expected Information as Expected Utility Bernardo, Jose M. The Annals of Statistics, Vol. 7, Issue 3 https://doi.org/10.1214/aos/1176344689	journal	May 1979
Bayes' Method for Bookies Freedman, David A.; Purves, Roger A. The Annals of Mathematical Statistics, Vol. 40, Issue 4 https://doi.org/10.1214/aoms/1177697494	journal	August 1969
Dynamic Coherence and Probability Kinematics Skyrms, Brian Philosophy of Science, Vol. 54, Issue 1 https://doi.org/10.1086/289350	journal	March 1987
Information Measures in Perspective: Information Measures in Perspective Ebrahimi, Nader; Soofi, Ehsan S.; Soyer, Refik International Statistical Review, Vol. 78, Issue 3 https://doi.org/10.1111/j.1751-5823.2010.00105.x	journal	December 2010
Gradient-based learning applied to document recognition Lecun, Y.; Bottou, L.; Bengio, Y. Proceedings of the IEEE, Vol. 86, Issue 11 https://doi.org/10.1109/5.726791	journal	January 1998
Capturing the Intangible Concept of Information Soofi, Ehsan S. Journal of the American Statistical Association, Vol. 89, Issue 428 https://doi.org/10.1080/01621459.1994.10476865	journal	December 1994
Sur une généralisation des intégrales de M. J. Radon Nikodym, Otton Fundamenta Mathematicae, Vol. 15 https://doi.org/10.4064/fm-15-1-131-179	journal	January 1930
Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy Shore, J.; Johnson, R. IEEE Transactions on Information Theory, Vol. 26, Issue 1 https://doi.org/10.1109/TIT.1980.1056144	journal	January 1980
On rational betting systems Adams, Ernest W. Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 3-4 https://doi.org/10.1007/bf01969549	journal	April 1964
Friendship stability in adolescence is associated with ventral striatum responses to vicarious rewards Schreuders, Elisabeth; Braams, Barbara R.; Crone, Eveline A. Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-020-20042-1	journal	January 2021
Skillful statistical models to predict seasonal wind speed and solar radiation in a Yangtze River estuary case study Zeng, Peng; Sun, Xun; Farnham, David J. Scientific Reports, Vol. 10, Issue 1 https://doi.org/10.1038/s41598-020-65281-w	journal	May 2020
A Mathematical Theory of Communication Shannon, C. E. Bell System Technical Journal, Vol. 27, Issue 4 https://doi.org/10.1002/j.1538-7305.1948.tb00917.x	journal	October 1948
Understanding predictive information criteria for Bayesian models Gelman, Andrew; Hwang, Jessica; Vehtari, Aki arXiv https://doi.org/10.48550/arxiv.1307.5928	preprint	January 2013
Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies Ilic, Velimir M.; Stankovic, Miomir S. arXiv https://doi.org/10.48550/arxiv.1311.0323	text	January 2013
Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies Jizba, Petr; Korbel, Jan arXiv https://doi.org/10.48550/arxiv.1808.01172	text	January 2018
Generalized statistics: yet another generalization Jizba, Petr; Arimitsu, Toshihico arXiv https://doi.org/10.48550/arxiv.cond-mat/0312034	text	January 2003
Information Properties of Order Statistics and Spacings Ebrahimi, N.; Soofi, E. S.; Zahedi, H. IEEE Transactions on Information Theory, Vol. 50, Issue 1 https://doi.org/10.1109/tit.2003.821973	journal	January 2004

Cited By (1)

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo Catanach, Thomas A.; Vo, Huy D.; Munsky, Brian arXiv https://doi.org/10.48550/arxiv.2001.01373	preprint	January 2020

Similar Records

On variational definition of quantum entropy

Journal Article · Tue Jan 13 00:00:00 EST 2015 · AIP Conference Proceedings · OSTI ID:1605735

Belavkin, Roman V.

Jensen-Shannon divergence as a measure of distinguishability between mixed quantum states

Journal Article · Tue Nov 15 00:00:00 EST 2005 · Physical Review. A · OSTI ID:1605735

Majtey, A P; Lamberti, P W; Prato, D P

Integrated analysis and consistency measurement of bremsstrahlung and charge exchange spectroscopy data for the determination of the ion effective charge

Journal Article · Fri Oct 15 00:00:00 EDT 2010 · Review of Scientific Instruments · OSTI ID:1605735

Verdoolaege, G; Fischer, R; Van Oost, G

Related Subjects

97 MATHEMATICS AND COMPUTING
information
Bayesian inference
entropy
self information
mutual information
Kullback–Leibler divergence
Lindley information
maximal uncertainty
proper utility

Title: Generalizing Information to the Evolution of Rational Belief

Citation Formats

References (37)

Cited By (1)

Similar Records

Related Subjects