skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Entropy and its Relationship with Statistics

Technical Report ·
DOI:· OSTI ID:1895025
 [1];  [1];  [1]
  1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

The purpose of our report is to discuss the notion of entropy and its relationship with statistics. Our goal is to provide a manner in which you can think about entropy, its central role within information theory and relationship with statistics. We review various relationships between information theory and statistics—nearly all are well-known but unfortunately are often not recognized. Entropy quantities the "average amount of surprise" in a random variable and lies at the heart of information theory, which studies the transmission, processing, extraction, and utilization of information. For us, data is information. What is the distinction between information theory and statistics? Information theorists work with probability distributions. Instead, statisticians work with samples. In so many words, information theory using samples is the practice of statistics.

Research Organization:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Organization:
USDOE National Nuclear Security Administration (NNSA); USDOE Laboratory Directed Research and Development (LDRD) Program
DOE Contract Number:
Report Number(s):
SAND2022-15094; 711399
Country of Publication:
United States

Similar Records

Obama Signs New START Treaty With Russians
Multimedia · 2010 · OSTI ID:1895025

Applied extreme-value statistics
Technical Report · 1983 · OSTI ID:1895025

"Test" is a Four Letter Word
Journal Article · 2005 · Better Software Magazine, N/A, N/A, October 1, 2005, N/A · OSTI ID:1895025

Related Subjects