How is entropy calculated from frequencies?

Each symbol's frequency is divided by the total frequency to obtain its probability. Shannon entropy is then calculated from those probabilities.

What is the unit of Shannon entropy?

Using logarithms to base 2, entropy is measured in bits.

Shannon Entropy Calculator

Q: What is Shannon entropy?

Shannon entropy measures the average uncertainty or information content of a probability distribution. It is calculated as the negative sum of p times log base 2 of p.

Calculate Shannon Entropy

Enter a probability distribution or symbol frequencies. Separate values with commas, semicolons, or new lines. You can optionally label each value using the format A: 10, B: 5, C: 1.

Input type

Probabilities must be between 0 and 1 and sum to 1.

Probabilities or frequencies

Up to 200 outcomes. Zero probabilities and zero frequencies are allowed.

Try an example

What Is Shannon Entropy?

Shannon entropy is a fundamental concept in information theory. Introduced by Claude Shannon, it measures the average uncertainty associated with the possible outcomes of a random variable. A distribution with equally likely outcomes has higher entropy than one in which a single outcome is much more likely.

Shannon entropy formula

H(X) = −Σ_i=1ⁿ p_i log₂(p_i)

H(X): entropy, measured in bits when using log base 2.
p_i: probability of outcome i.
n: number of outcomes in the supplied distribution.
log₂: logarithm to base 2.

By convention, a zero-probability outcome contributes zero because the limit of p log₂(p) as p approaches zero is zero.

Example 1: Fair coin

A fair coin has two outcomes, each with probability 0.5.

H(X) = −[0.5 log₂(0.5) + 0.5 log₂(0.5)] = 1 bit

Both outcomes are equally likely, so the entropy is one bit.

Example 2: Unequal probabilities

Consider a distribution with probabilities 0.7, 0.2, and 0.1.

H(X) = −[0.7 log₂(0.7) + 0.2 log₂(0.2) + 0.1 log₂(0.1)] ≈ 1.1568 bits

This is below the maximum possible entropy for three outcomes, log₂(3) ≈ 1.585 bits, because the outcomes are not equally likely.

Calculating entropy from symbol frequencies

Frequencies are converted to probabilities using:

p_i = f_i / Σ f_j

For example, if a dataset contains A = 10, B = 5, and C = 1, the total is 16. The probabilities are 10/16, 5/16, and 1/16. The calculator then applies the Shannon formula to those probabilities.

Maximum and normalized entropy

For N possible outcomes, the maximum entropy is log₂(N), attained when all N outcomes are equally likely. Normalized entropy divides the calculated entropy by this maximum, giving a value between zero and one for a valid distribution with at least two outcomes.

Where is Shannon entropy used?

Data compression: estimating the theoretical information limit for lossless coding.
Probability and statistics: describing uncertainty in a distribution.
Machine learning: evaluating class distributions and decision-tree splits.
Communications: studying information sources and channel capacity.
Linguistics: examining the distribution of letters, words, or symbols.

Frequently Asked Questions

What does Shannon entropy measure?

It measures the average uncertainty in a probability distribution. Higher entropy means outcomes are, on average, less predictable. Lower entropy means the distribution is more concentrated on particular outcomes.

Why is entropy measured in bits?

This calculator uses logarithms to base 2. Base-2 entropy is expressed in bits. Using a different logarithm base would change the unit and numerical value.

Can I calculate entropy from frequencies instead of probabilities?

Yes. Select symbol frequencies and enter non-negative whole-number counts. The calculator converts each count to a probability by dividing it by the total count.

Do probabilities have to sum to one?

Yes. In probability mode, the entered values must sum to 1. If you have counts or frequencies, use frequency mode instead.

What is the entropy of a certain outcome?

If one outcome has probability 1 and all other outcomes have probability 0, the entropy is zero bits. There is no uncertainty about which outcome will occur.

Is Shannon entropy the same as cryptographic hash entropy?

Shannon entropy is a mathematical measure of uncertainty in a probability distribution. It is not a hash function and does not generate hashes. It can be used to analyze symbol distributions, but entropy alone does not establish cryptographic security.