On this page
Every photo you send, every video call and every file on your phone rests on one idea: information can be measured, like mass or charge. The person who showed how was Claude Shannon, an American mathematician and engineer. This post explains what he measured, works through an example taken from his own paper, and ends with what a student can borrow from the way he worked.
Who was Claude Shannon?
Shannon was born on 30 April 1916 in Petoskey, Michigan, and grew up in nearby Gaylord, according to MIT's obituary. He took degrees in mathematics and electrical engineering at the University of Michigan in 1936, then went to MIT, where he worked part-time on Vannevar Bush's differential analyser, an early mechanical computer.
His MIT master's thesis, A Symbolic Analysis of Relay and Switching Circuits, showed that Boolean algebra, the algebra of true and false, or 1 and 0, could describe and simplify circuits built from relays. MIT's obituary calls this "the beginning of modern switching theory". He joined Bell Laboratories in 1941 and stayed until 1972, later teaching at MIT. He died on 24 February 2001.
What problem did Shannon solve in 1948?
Engineers in the 1940s could send signals down wires and through the air, but they had no single measure of how much a channel could carry. Shannon's paper A Mathematical Theory of Communication appeared in two parts in the Bell System Technical Journal in July and October 1948.
Its opening states the problem: "The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point." He then set meaning aside. What matters to an engineer, he wrote, is that the message is one selection from a set of possible messages.
That move is the heart of the theory. If a message is one choice out of N equally likely choices, the information it carries grows with the logarithm of N. Shannon chose base 2, so the unit is the binary digit, which, as he wrote, is "a word suggested by J. W. Tukey": the bit. One relay or one switch with two states stores one bit. N switches have 2^N states and store N bits.
How does Shannon's entropy formula work?
When the choices are not equally likely, Shannon's measure is the entropy:
H = −(p1 log2 p1 + p2 log2 p2 + ... + pn log2 pn)
where each p is the probability of one symbol. The paper shows that H is zero only when one outcome is certain, and is largest, equal to log2 n, when all n outcomes are equally likely.
Worked example 1: a fair coin and a biased coin.
Fair coin, p = 1/2 each: H = −(1/2 × (−1) + 1/2 × (−1)) = 1 bit per toss.
Coin that lands heads 90% of the time: log2 0.9 ≈ −0.152 and log2 0.1 ≈ −3.322. So H ≈ 0.9 × 0.152 + 0.1 × 3.322 ≈ 0.137 + 0.332 = 0.469 bits per toss.
The biased coin is more predictable, so each toss tells you less.
Worked example 2: Shannon's own four-letter source. The paper takes symbols A, B, C and D with probabilities 1/2, 1/4, 1/8 and 1/8.
H = 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits per symbol. (Each term is p × log2(1/p).)
A plain code needs 2 binary digits per symbol (00, 01, 10, 11).
Shannon gives the code A = 0, B = 10, C = 110, D = 111. Average length = 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 1.75 digits.
The clever code matches the entropy exactly. Common letters get short codes, rare ones get long codes, and no code is the start of another, so a string like 010110 can be read back without commas: A, B, C.
Why does this matter for a student today?
Shannon also showed that adding carefully chosen extra bits lets a receiver correct errors caused by noise, an idea behind the error-correcting codes in modern phones and storage. MacTutor's biography describes this result, and MIT's obituary adds that his 1949 paper on secrecy systems is generally credited with turning cryptography from an art into a science. If you go on to computer science, you will meet his entropy formula again in data compression and machine learning, and the same p log p form appears in statistical physics. For the story of the other founder of computing in this era, read our post on Alan Turing and computing.
What can you learn from how Shannon worked?
Strip a problem to its core. He threw out meaning on page one, and that made the problem solvable.
Borrow tools from other subjects. His master's thesis took an algebra from logic and applied it to wiring.
Check ideas on a tiny case. The four-letter source is small enough to work by hand, yet it shows the whole theory.
Play. MIT's obituary lists his juggling machine and a mechanical mouse that could find its way through a maze.
If logarithms are the part that slows you down, the Fermi Concept Library covers every concept in mathematics and computing with its own lesson, so you can revise log rules before trying entropy questions.
Try it now: work out the entropy of a fair six-sided die, then check that your answer equals log2 6, about 2.585 bits.
