MathLiveInformation entropy
Gives the average amount of information, or uncertainty, of a discrete random variable as the probability-weighted sum of the information content of its outcomes. It is used in information theory, data compression and statistics to measure how unpredictable a source is.
H(X) = \sum_{i=1}^{n} p(x_i) \, I(x_i)What the symbols mean
| information entropy of the random variable X, in bits if the logarithm is base 2 | |
| number of possible outcomes of X | |
| the i-th possible outcome of X | |
| probability of outcome x_i | |
| information content of outcome x_i, typically -\log p(x_i) |
About this formula
- FieldComputer science
- TopicInformation theory
- LevelUniversity
- Also calledShannon entropy, entropy, average information content, expected information, entropy of a random variable, Shannon information entropy formula
- SourceWikidata Q204570
Write it in Obsidian
MathLive is a free visual math editor for Obsidian. Run Find a formula, type “information entropy”, and it goes into your note as editable LaTeX. You can also scan formulas from screenshots, PDFs and handwriting.