Published
In this lesson
Wide tables, equations, and code scroll sideways. Swipe, or Tab to focus them and use the left and right arrow keys.
Shannon entropy
You will learn: Measure expected surprise and connect it with finite-label uncertainty and code lengths.
Start with: Reading a distribution · Expectation and variance
How much surprise should you expect before seeing an outcome? Start with probability mass and weighted expectations. This lesson measures uncertainty in a specified finite distribution. It does not estimate uncertainty from a dataset.
An outcome with probability p carries surprise −log₂ p bits. A certain outcome carries zero; halving a positive probability adds one bit. Independent outcomes multiply their probabilities, so their surprises add. Averaging over the possible outcomes gives entropy:
We define 0 log 0 as zero by its limit. An impossible outcome is never drawn from P; it adds no expected surprise.
Set relative weights for three labeled outcomes. Each vector is normalized by its own total. They specify probabilities, not collected observations. An all-zero vector is invalid and is never replaced with a uniform distribution.
Filled blue bars: P, the distribution used for averaging. Both use the same outcome labels.
| Quantity | bits per outcome |
|---|---|
| Entropy H(P) | 1.500000 |
| Maximum entropy over three labels | 1.584963 |
Each outcome contributes to entropy
| Outcome | P(outcome) | −p log p |
|---|---|---|
| A | 0.5000000 | 0.5000000 |
| B | 0.2500000 | 0.5000000 |
| C | 0.2500000 | 0.5000000 |
An outcome with source probability zero contributes zero, including when its prediction probability is also zero. At certainty the only occurring outcome has zero surprise, so entropy is zero.
A code you can inspect
The default weights 2,1,1 give probabilities 1/2,1/4,1/4. Their surprises are 1,2,2 bits and their entropy is 1.5 bits. Encode A as 0, B as 10, and C as 11. No codeword begins with another complete codeword, so a concatenated message can be decoded unambiguously.
For example, ABCA becomes 010110: six bits for four symbols. That one message happens to average 1.5 bits per symbol. Other messages differ: AAAA takes four bits and BBBB takes eight. The expected length is 0.5·1+0.25·2+0.25·2=1.5; entropy does not prescribe the length of every message.
For general probabilities, −log₂ p need not be an integer. Entropy lower-bounds expected length for binary prefix codes, while a Shannon code can achieve a length less than H+1. Encoding independent, identically distributed symbols in longer blocks reduces this per-symbol overhead. Dependence requires a model of sequences, rather than simply multiplying the one-symbol entropy by message length.
Make a prediction
Does entropy 1.5 bits require a codeword containing half a bit?
Explore the answer
No. Codewords have whole-number lengths. Their probability-weighted average can be fractional, just as an expected count can be fractional even though every observed count is an integer.
What changes uncertainty?
Set all three weights equal: entropy rises to log₂ 3≈1.584963 bits. Set two weights to zero: the remaining outcome is certain and entropy is zero. These are the maximum and minimum on three labels. The maximum follows from nonnegative KL divergence: for the uniform U, KL(P∥U)=log₂ 3−H(P).
Multiplying every weight by the same positive number does nothing after normalization. Renaming A, B, C also does nothing if probabilities move with their labels. Merging B and C is different: the default three-outcome model becomes a fair binary model with entropy one bit. The removed half bit was uncertainty about B versus C on the half of draws that were not A.
Switching to natural logarithms multiplies every bit value by ln 2. The default becomes approximately 1.039721 nats. This changes units, not the ordering of distributions by entropy.
Probability model, sample, and measurement scale
If you substitute observed relative frequencies for P, you compute the entropy of the empirical distribution. It need not equal population entropy, and an unseen outcome need not be impossible. The controls here define P directly so that these two interpretations cannot be confused.
The finite-label formula is not a coordinate-free formula for continuous measurements. Differential entropy uses a density and can be negative; changing measurement units changes it. Discretizing a continuous measurement also makes the chosen bins part of the question.
Make a prediction
Will a perfect probability prediction make entropy zero?
Explore the answer
Only if the source itself is certain. Predicting the default probabilities exactly leaves 1.5 bits of expected surprise. The next lesson separates that source uncertainty from the extra loss caused by predicting the wrong probabilities.
Continue to cross-entropy, then KL divergence, using the same three labeled outcomes.
Sources
CMU’s source-coding notes give the prefix-code bounds and independent-block argument. SciPy’s entropy reference documents the discrete formula, normalization, and units. The explicit code, message lengths, and category-merging example above are directly checkable calculations.