Generalizing Information to the Evolution of Rational Belief

Duersch, Jed A.; Catanach, Thomas A.

doi:10.3390/e22010108

Title: Generalizing Information to the Evolution of Rational Belief

Abstract

Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome’s plausibility. Information measures based on Shannon’s concept of entropy include realization information, Kullback–Leibler divergence, Lindley’s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.

Authors:

^[1]; Catanach, Thomas A. ^[1]

Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Publication Date:: Thu Jan 16 00:00:00 EST 2020

Research Org.:: Sandia National Lab. (SNL-CA), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1605735

Alternate Identifier(s):: OSTI ID: 1618088

Report Number(s):: SAND-2020-3103J; SAND-2019-13298J
Journal ID: ISSN 1099-4300; ENTRFG; 684733

Grant/Contract Number:: AC04-94AL85000

Resource Type:: Accepted Manuscript

Journal Name:: Entropy

Additional Journal Information:: Journal Volume: 22; Journal Issue: 1; Journal ID: ISSN 1099-4300

Publisher:: MDPI

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; information; Bayesian inference; entropy; self information; mutual information; Kullback–Leibler divergence; Lindley information; maximal uncertainty; proper utility

Citation Formats


                    Duersch, Jed A., and Catanach, Thomas A. Generalizing Information to the Evolution of Rational Belief.  United States: N. p., 2020. 
Web.  doi:10.3390/e22010108.

Copy to clipboard


                    Duersch, Jed A., & Catanach, Thomas A. Generalizing Information to the Evolution of Rational Belief.  United States.  https://doi.org/10.3390/e22010108

Copy to clipboard


                    Duersch, Jed A., and Catanach, Thomas A. Thu .  
"Generalizing Information to the Evolution of Rational Belief".  United States.  https://doi.org/10.3390/e22010108.  https://www.osti.gov/servlets/purl/1605735.

Copy to clipboard


                    
@article{osti_1605735,

  title        = {Generalizing Information to the Evolution of Rational Belief},

  author       = {Duersch, Jed A. and Catanach, Thomas A.},

  abstractNote = {Information theory provides a mathematical foundation to measure uncertainty in belief. Belief is represented by a probability distribution that captures our understanding of an outcome’s plausibility. Information measures based on Shannon’s concept of entropy include realization information, Kullback–Leibler divergence, Lindley’s information in experiment, cross entropy, and mutual information. We derive a general theory of information from first principles that accounts for evolving belief and recovers all of these measures. Rather than simply gauging uncertainty, information is understood in this theory to measure change in belief. We may then regard entropy as the information we expect to gain upon realization of a discrete latent random variable. This theory of information is compatible with the Bayesian paradigm in which rational belief is updated as evidence becomes available. Furthermore, this theory admits novel measures of information with well-defined properties, which we explored in both analysis and experiment. This view of information illuminates the study of machine learning by allowing us to quantify information captured by a predictive model and distinguish it from residual information contained in training data. We gain related insights regarding feature selection, anomaly detection, and novel Bayesian approaches.},

  doi          = {10.3390/e22010108},

  journal      = {Entropy},

  number       = 1,

  volume       = 22,

  place        = {United States},

  year         = {Thu Jan 16 00:00:00 EST 2020},

  month        = {Thu Jan 16 00:00:00 EST 2020}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.3390/e22010108

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 4 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

A Mathematical Theory of Communication
journal, July 1948

Shannon, C. E.
Bell System Technical Journal, Vol. 27, Issue 3
DOI: 10.1002/j.1538-7305.1948.tb01338.x

Estimating the Dimension of a Model
journal, March 1978

Schwarz, Gideon
The Annals of Statistics, Vol. 6, Issue 2
DOI: 10.1214/aos/1176344136

The Theory of Information
journal, January 1951

Barnard, G. A.
Journal of the Royal Statistical Society: Series B (Methodological), Vol. 13, Issue 1
DOI: 10.1111/j.2517-6161.1951.tb00069.x

On rational betting systems
journal, March 1962

Adams, Ernest W.
Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 1-2
DOI: 10.1007/BF02025803

A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions
journal, January 2011

Hanel, R.; Thurner, S.
EPL (Europhysics Letters), Vol. 93, Issue 2
DOI: 10.1209/0295-5075/93/20006

On confirmation and rational betting
journal, September 1955

Lehman, R. Sherman
Journal of Symbolic Logic, Vol. 20, Issue 3
DOI: 10.2307/2268221

On the Distribution Function of Additive Functions
journal, January 1946

Erdos, P.
The Annals of Mathematics, Vol. 47, Issue 1
DOI: 10.2307/1969031

Kullback-Leibler information as a basis for strong inference in ecological studies
journal, January 2001

Burnham, Kenneth P.; Anderson, David R.
Wildlife Research, Vol. 28, Issue 2
DOI: 10.1071/WR99107

Information Theory and Statistical Mechanics
journal, May 1957

Jaynes, E. T.
Physical Review, Vol. 106, Issue 4
DOI: 10.1103/PhysRev.106.620

Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables
journal, September 1963

Good, I. J.
The Annals of Mathematical Statistics, Vol. 34, Issue 3
DOI: 10.1214/aoms/1177704014

Group entropies, correlation laws, and zeta functions
journal, August 2011

Tempesta, Piergiulio
Physical Review E, Vol. 84, Issue 2
DOI: 10.1103/PhysRevE.84.021121

A new look at the statistical model identification
journal, December 1974

Akaike, H.
IEEE Transactions on Automatic Control, Vol. 19, Issue 6
DOI: 10.1109/TAC.1974.1100705

Generalized Shannon–Khinchin axioms and uniqueness theorem for pseudo-additive entropies
journal, October 2014

Ilić, Velimir M.; Stanković, Miomir S.
Physica A: Statistical Mechanics and its Applications, Vol. 411
DOI: 10.1016/j.physa.2014.05.009

On Information and Sufficiency
journal, March 1951

Kullback, S.; Leibler, R. A.
The Annals of Mathematical Statistics, Vol. 22, Issue 1
DOI: 10.1214/aoms/1177729694

Properties of cross-entropy minimization
journal, July 1981

Shore, J.; Johnson, R.
IEEE Transactions on Information Theory, Vol. 27, Issue 4
DOI: 10.1109/TIT.1981.1056373

On a Measure of the Information Provided by an Experiment
journal, December 1956

Lindley, D. V.
The Annals of Mathematical Statistics, Vol. 27, Issue 4
DOI: 10.1214/aoms/1177728069

Understanding predictive information criteria for Bayesian models
journal, August 2013

Gelman, Andrew; Hwang, Jessica; Vehtari, Aki
Statistics and Computing, Vol. 24, Issue 6
DOI: 10.1007/s11222-013-9416-2

Probability, Frequency and Reasonable Expectation
journal, January 1946

Cox, R. T.
American Journal of Physics, Vol. 14, Issue 1
DOI: 10.1119/1.1990764

Principal Information Theoretic Approaches
journal, December 2000

Soofi, Ehsan S.
Journal of the American Statistical Association, Vol. 95, Issue 452
DOI: 10.1080/01621459.2000.10474346

Generalized statistics: yet another generalization
journal, September 2004

Jizba, Petr; Arimitsu, Toshihico
Physica A: Statistical Mechanics and its Applications, Vol. 340, Issue 1-3
DOI: 10.1016/j.physa.2004.03.085

Expected Information as Expected Utility
journal, May 1979

Bernardo, Jose M.
The Annals of Statistics, Vol. 7, Issue 3
DOI: 10.1214/aos/1176344689

Bayes' Method for Bookies
journal, August 1969

Freedman, David A.; Purves, Roger A.
The Annals of Mathematical Statistics, Vol. 40, Issue 4
DOI: 10.1214/aoms/1177697494

Dynamic Coherence and Probability Kinematics
journal, March 1987

Skyrms, Brian
Philosophy of Science, Vol. 54, Issue 1
DOI: 10.1086/289350

Information Measures in Perspective: Information Measures in Perspective
journal, December 2010

Ebrahimi, Nader; Soofi, Ehsan S.; Soyer, Refik
International Statistical Review, Vol. 78, Issue 3
DOI: 10.1111/j.1751-5823.2010.00105.x

Gradient-based learning applied to document recognition
journal, January 1998

Lecun, Y.; Bottou, L.; Bengio, Y.
Proceedings of the IEEE, Vol. 86, Issue 11
DOI: 10.1109/5.726791

Capturing the Intangible Concept of Information
journal, December 1994

Soofi, Ehsan S.
Journal of the American Statistical Association, Vol. 89, Issue 428
DOI: 10.1080/01621459.1994.10476865

Sur une généralisation des intégrales de M. J. Radon
journal, January 1930

Nikodym, Otton
Fundamenta Mathematicae, Vol. 15
DOI: 10.4064/fm-15-1-131-179

Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
journal, January 1980

Shore, J.; Johnson, R.
IEEE Transactions on Information Theory, Vol. 26, Issue 1
DOI: 10.1109/TIT.1980.1056144

On rational betting systems
journal, April 1964

Adams, Ernest W.
Archiv für Mathematische Logik und Grundlagenforschung, Vol. 6, Issue 3-4
DOI: 10.1007/bf01969549

Friendship stability in adolescence is associated with ventral striatum responses to vicarious rewards
journal, January 2021

Schreuders, Elisabeth; Braams, Barbara R.; Crone, Eveline A.
Nature Communications, Vol. 12, Issue 1
DOI: 10.1038/s41467-020-20042-1

Skillful statistical models to predict seasonal wind speed and solar radiation in a Yangtze River estuary case study
journal, May 2020

Zeng, Peng; Sun, Xun; Farnham, David J.
Scientific Reports, Vol. 10, Issue 1
DOI: 10.1038/s41598-020-65281-w

A Mathematical Theory of Communication
journal, October 1948

Shannon, C. E.
Bell System Technical Journal, Vol. 27, Issue 4
DOI: 10.1002/j.1538-7305.1948.tb00917.x

Understanding predictive information criteria for Bayesian models
preprint, January 2013

Gelman, Andrew; Hwang, Jessica; Vehtari, Aki
arXiv
DOI: 10.48550/arxiv.1307.5928

Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies
text, January 2013

Ilic, Velimir M.; Stankovic, Miomir S.
arXiv
DOI: 10.48550/arxiv.1311.0323

Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies
text, January 2018

Jizba, Petr; Korbel, Jan
arXiv
DOI: 10.48550/arxiv.1808.01172

Generalized statistics: yet another generalization
text, January 2003

Jizba, Petr; Arimitsu, Toshihico
arXiv
DOI: 10.48550/arxiv.cond-mat/0312034

Principal Information Theoretic Approaches
journal, December 2000

Soofi, Ehsan S.
Journal of the American Statistical Association, Vol. 95, Issue 452
DOI: 10.1080/01621459.2000.10474346

Information Properties of Order Statistics and Spacings
journal, January 2004

Ebrahimi, N.; Soofi, E. S.; Zahedi, H.
IEEE Transactions on Information Theory, Vol. 50, Issue 1
DOI: 10.1109/tit.2003.821973

On a Measure of the Information Provided by an Experiment
journal, December 1956

Lindley, D. V.
The Annals of Mathematical Statistics, Vol. 27, Issue 4
DOI: 10.1214/aoms/1177728069

On the Distribution Function of Additive Functions
journal, January 1946

Erdos, P.
The Annals of Mathematics, Vol. 47, Issue 1
DOI: 10.2307/1969031

Sur une généralisation des intégrales de M. J. Radon
journal, January 1930

Nikodym, Otton
Fundamenta Mathematicae, Vol. 15
DOI: 10.4064/fm-15-1-131-179

Works referencing / citing this record:

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo
preprint, January 2020

Catanach, Thomas A.; Vo, Huy D.; Munsky, Brian
arXiv
DOI: 10.48550/arxiv.2001.01373

Similar Records in DOE PAGES and OSTI.GOV collections:

On variational definition of quantum entropy

Journal Article Belavkin, Roman V. - AIP Conference Proceedings

Entropy of distribution P can be defined in at least three different ways: 1) as the expectation of the Kullback-Leibler (KL) divergence of P from elementary δ-measures (in this case, it is interpreted as expected surprise); 2) as a negative KL-divergence of some reference measure ν from the probability measure P; 3) as the supremum of Shannon’s mutual information taken over all channels such that P is the output probability, in which case it is dual of some transportation problem. In classical (i.e. commutative) probability, all three definitions lead to the same quantity, providing only different interpretations of entropy. Inmore »« less
https://doi.org/10.1063/1.4905979
Jensen-Shannon divergence as a measure of distinguishability between mixed quantum states

Journal Article Majtey, A P ; Lamberti, P W ; Prato, D P - Physical Review. A

We discuss an alternative to relative entropy as a measure of distance between mixed quantum states. The proposed quantity is an extension to the realm of quantum theory of the Jensen-Shannon divergence (JSD) between probability distributions. The JSD has several interesting properties. It arises in information theory and, unlike the Kullback-Leibler divergence, it is symmetric, always well-defined, and bounded. We show that the quantum JSD shares with the relative entropy most of the physically relevant properties, in particular those required for a 'good' quantum distinguishability measure. We relate it to other known quantum distances and we suggest possible applications inmore »« less
https://doi.org/10.1103/PHYSREVA.72.0
Integrated analysis and consistency measurement of bremsstrahlung and charge exchange spectroscopy data for the determination of the ion effective charge

Journal Article Verdoolaege, G ; Fischer, R ; Van Oost, G - Review of Scientific Instruments

In the context of Bayesian probability theory, we discuss a model for estimating the plasma ion effective charge Z{sub eff}, integrating data from both bremsstrahlung spectroscopy and individual impurity concentrations obtained via charge exchange spectroscopy (CXS). The validity of the model, taking into account statistical as well as systematic uncertainties, is shown via the deviance information criterion. The consistency of the continuum and CXS data regarding Z{sub eff} is improved, as measured by the symmetrized Kullback-Leibler divergence and the geodesic distance between the respective Z{sub eff} marginal posterior densities.
https://doi.org/10.1063/1.3464466
Sensitivity Analysis and Assessment of Prior Model Probabilities in MLBMA with Application to Unsaturated Fractured Tuff

Journal Article Ye, Ming ; Neuman, Shlomo P ; Meyer, Philip D ; ... - Water Resources Research

Previous application of Maximum Likelihood Bayesian Model Averaging (MLBMA, Neuman [2002, 2003]) to alternative variogram models of log air permeability data in fractured tuff has demonstrated its effectiveness in quantifying conceptual model uncertainty and enhancing predictive capability [Ye et al., 2004]. A question remained how best to ascribe prior probabilities to competing models. In this paper we examine the extent to which lead statistics of posterior log permeability predictions are sensitive to prior probabilities of seven corresponding variogram models. We then explore the feasibility of quantifying prior model probabilities by (a) maximizing Shannon's entropy H [Shannon, 1948] subject to constraintsmore »« less
https://doi.org/10.1029/2005WR004260
Learning Uncertainties the Frequentist Way: Calibration and Correlation in High Energy Physics

Journal Article Gambhir, Rikab ; Nachman, Benjamin ; Thaler, Jesse - Physical Review Letters

Calibration is a common experimental physics problem, whose goal is to infer the value and uncertainty of an unobservable quantity Z given a measured quantity X. Additionally, one would like to quantify the extent to which X and Z are correlated. In this Letter, we present a machine learning framework for performing frequentist maximum likelihood inference with Gaussian uncertainty estimation, which also quantifies the mutual information between the unobservable and measured quantities. This framework uses the Donsker-Varadhan representation of the Kullback-Leibler divergence—parametrized with a novel Gaussian ansatz—to enable a simultaneous extraction of the maximum likelihood values, uncertainties, and mutual informationmore »« less
https://doi.org/10.1103/PhysRevLett.129.082001

Similar Records

Title: Generalizing Information to the Evolution of Rational Belief

Abstract

Citation Formats

A Mathematical Theory of Communication journal, July 1948

Estimating the Dimension of a Model journal, March 1978

The Theory of Information journal, January 1951

On rational betting systems journal, March 1962

A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions journal, January 2011

On confirmation and rational betting journal, September 1955

On the Distribution Function of Additive Functions journal, January 1946

Kullback-Leibler information as a basis for strong inference in ecological studies journal, January 2001

Information Theory and Statistical Mechanics journal, May 1957

Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables journal, September 1963

Group entropies, correlation laws, and zeta functions journal, August 2011

A new look at the statistical model identification journal, December 1974

Generalized Shannon–Khinchin axioms and uniqueness theorem for pseudo-additive entropies journal, October 2014

On Information and Sufficiency journal, March 1951

Properties of cross-entropy minimization journal, July 1981

On a Measure of the Information Provided by an Experiment journal, December 1956

Understanding predictive information criteria for Bayesian models journal, August 2013

Probability, Frequency and Reasonable Expectation journal, January 1946

Principal Information Theoretic Approaches journal, December 2000

Generalized statistics: yet another generalization journal, September 2004

Expected Information as Expected Utility journal, May 1979

Bayes' Method for Bookies journal, August 1969

Dynamic Coherence and Probability Kinematics journal, March 1987

Information Measures in Perspective: Information Measures in Perspective journal, December 2010

Gradient-based learning applied to document recognition journal, January 1998

Capturing the Intangible Concept of Information journal, December 1994

Sur une généralisation des intégrales de M. J. Radon journal, January 1930

Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy journal, January 1980

On rational betting systems journal, April 1964

Friendship stability in adolescence is associated with ventral striatum responses to vicarious rewards journal, January 2021

Skillful statistical models to predict seasonal wind speed and solar radiation in a Yangtze River estuary case study journal, May 2020

A Mathematical Theory of Communication journal, October 1948

Understanding predictive information criteria for Bayesian models preprint, January 2013

Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies text, January 2013

Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies text, January 2018

Generalized statistics: yet another generalization text, January 2003

Principal Information Theoretic Approaches journal, December 2000

Information Properties of Order Statistics and Spacings journal, January 2004

On a Measure of the Information Provided by an Experiment journal, December 1956

On the Distribution Function of Additive Functions journal, January 1946

Sur une généralisation des intégrales de M. J. Radon journal, January 1930

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo preprint, January 2020

A Mathematical Theory of Communication
journal, July 1948

Estimating the Dimension of a Model
journal, March 1978

The Theory of Information
journal, January 1951

On rational betting systems
journal, March 1962

A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions
journal, January 2011

On confirmation and rational betting
journal, September 1955

On the Distribution Function of Additive Functions
journal, January 1946

Kullback-Leibler information as a basis for strong inference in ecological studies
journal, January 2001

Information Theory and Statistical Mechanics
journal, May 1957

Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables
journal, September 1963

Group entropies, correlation laws, and zeta functions
journal, August 2011

A new look at the statistical model identification
journal, December 1974

Generalized Shannon–Khinchin axioms and uniqueness theorem for pseudo-additive entropies
journal, October 2014

On Information and Sufficiency
journal, March 1951

Properties of cross-entropy minimization
journal, July 1981

On a Measure of the Information Provided by an Experiment
journal, December 1956

Understanding predictive information criteria for Bayesian models
journal, August 2013

Probability, Frequency and Reasonable Expectation
journal, January 1946

Principal Information Theoretic Approaches
journal, December 2000

Generalized statistics: yet another generalization
journal, September 2004

Expected Information as Expected Utility
journal, May 1979

Bayes' Method for Bookies
journal, August 1969

Dynamic Coherence and Probability Kinematics
journal, March 1987

Information Measures in Perspective: Information Measures in Perspective
journal, December 2010

Gradient-based learning applied to document recognition
journal, January 1998

Capturing the Intangible Concept of Information
journal, December 1994

Sur une généralisation des intégrales de M. J. Radon
journal, January 1930

Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
journal, January 1980

On rational betting systems
journal, April 1964

Friendship stability in adolescence is associated with ventral striatum responses to vicarious rewards
journal, January 2021

Skillful statistical models to predict seasonal wind speed and solar radiation in a Yangtze River estuary case study
journal, May 2020

A Mathematical Theory of Communication
journal, October 1948

Understanding predictive information criteria for Bayesian models
preprint, January 2013

Generalized Shannon-Khinchin Axioms and Uniqueness Theorem for Pseudo-additive Entropies
text, January 2013

Maximum Entropy Principle in statistical inference: case for non-Shannonian entropies
text, January 2018

Generalized statistics: yet another generalization
text, January 2003

Principal Information Theoretic Approaches
journal, December 2000

Information Properties of Order Statistics and Spacings
journal, January 2004

On a Measure of the Information Provided by an Experiment
journal, December 1956

On the Distribution Function of Additive Functions
journal, January 1946

Sur une généralisation des intégrales de M. J. Radon
journal, January 1930

Bayesian inference of Stochastic reaction networks using Multifidelity Sequential Tempered Markov Chain Monte Carlo
preprint, January 2020