On this page
Every time a survey reports a percentage or a lab reports a measured value, someone has made an estimate from limited data. Two questions follow. How precise can that estimate possibly be? And can a given estimate be improved? C. R. Rao answered both in one paper in 1945. This post explains his results with a coin you can toss yourself.
Who was C. R. Rao?
Calyampudi Radhakrishna Rao spent 40 years at the Indian Statistical Institute, founded by P. C. Mahalanobis, whose story we tell in our post on Mahalanobis and Indian statistics. According to the University at Buffalo obituary, he earned an MA in mathematics from Andhra University, an MA in statistics from Calcutta University and a PhD from Cambridge in 1948. After retiring from ISI in 1979 he taught in the United States, joined Buffalo in 2010, and died on 22 August 2023 at 102. He received the US National Medal of Science in 2002 and supervised 51 PhD students.
The Royal Statistical Society announced in April 2023 that he had won the International Prize in Statistics for his 1945 paper in the Bulletin of the Calcutta Mathematical Society, which the RSS says contained three results: the Cramer-Rao lower bound, the Rao-Blackwell theorem, and ideas that started the field now called information geometry. The paper is reprinted in the collection Breakthroughs in Statistics. The bound also carries the name of the Swedish statistician Harald Cramer.
What problem do these results solve?
Suppose a coin lands heads with an unknown probability p. You toss it n times and want to estimate p. An estimator is any rule that turns your data into a guess. It is unbiased if, averaged over many repeats of the experiment, it gives the true p. Among unbiased estimators, the one with the smallest variance (spread around the true value) is the most precise. Our guide to mean and variance covers the basics.
The RSS describes the Cramer-Rao bound as "a means for knowing when a method for estimating a quantity is as good as any method can be", and the Rao-Blackwell theorem as "a means for transforming an estimate into a better – in fact, an optimal – estimate".
A worked example: the Cramer-Rao bound for a coin
Each toss gives X = 1 (heads) with probability p, or X = 0 with probability 1 − p.
Fisher information. This measures how much one observation tells you about p. For a single toss it works out to I(p) = 1 / [p(1 − p)].
The bound. For n independent tosses, no unbiased estimator can have variance below 1 / [n I(p)] = p(1 − p) / n.
Numbers. Take a fair coin, p = 0.5, and n = 100 tosses. The bound is 0.5 × 0.5 / 100 = 0.0025, so the standard deviation can be no smaller than √0.0025 = 0.05.
The obvious estimator. The fraction of heads, (X₁ + ... + Xₙ) / n, has variance p(1 − p) / n, exactly the bound. So no unbiased method can beat simply counting heads.
That is the practical use: once your estimator reaches the bound, you can stop searching for a better one. If it does not, you know there is room to improve.
A worked example: Rao-Blackwell improves a rough estimate
Start with a deliberately poor but unbiased estimator: use only the first toss, X₁. Its average is p, so it is unbiased, but its variance is p(1 − p) = 0.25 for a fair coin. That is a hundred times worse than the bound.
The Rao-Blackwell theorem says: take your estimator and average it over everything consistent with a sufficient statistic, a summary of the data that holds all the information about p. For coin tosses, the total number of heads T is sufficient.
Ask: given that T heads appeared in n tosses, what is the expected value of X₁?
By symmetry, any one toss is equally likely to be any of the n tosses, so the chance the first was heads is T/n.
The improved estimator is T/n, the fraction of heads, with variance 0.0025.
The rough estimator became the optimal one. Blackwell's 1947 paper in the Annals of Mathematical Statistics states the same result: conditioning an unbiased estimate on a sufficient statistic gives an unbiased estimate with smaller variance.
Where are these ideas used today?
The RSS lists applications of the Cramer-Rao bound in signal processing, spectroscopy, radar, risk analysis and quantum physics, and of Rao-Blackwellisation in particle filtering and econometrics. In a physics lab, the bound tells you the smallest error bar your data can support. Our post on error propagation in measurements shows the school-level version of the same question.
What can a student take from his way of working?
Ask what the limit is. Rao did not only build a method; he asked how good any method could be. That question is useful in exams too: what is the fastest possible route to this answer?
Start from a bad answer and improve it. Rao-Blackwell is a recipe for upgrading a rough idea, which is how most problem solving actually goes.
Work with simple cases first. Coin tosses carry the whole idea. Test every new theorem you learn on the simplest example you can find.
If probability notation still slows you down, the Fermi Concept Library has every concept in mathematics with its own lesson and its place in the map.
Toss a coin 20 times, write down the fraction of heads, and work out the Cramer-Rao bound for your own n.
