On this page
A test for a rare disease is 99 percent accurate, and you test positive. Most people would say you are 99 percent likely to have it. The true answer can be closer to 9 percent. That gap is the base rate fallacy.
What is the base rate fallacy?
The base rate fallacy is the mistake of judging a probability from new evidence, such as a test result, while ignoring the base rate: how common the thing was before you looked. In medicine the base rate is the prevalence, the share of people who have the disease.
A test result does not replace the base rate; it updates it. When a disease is rare, even a small error rate produces more false alarms than true cases, because there are far more healthy people to make mistakes on.
Why can a positive result from a 99% accurate test be mostly wrong?
Count people instead of juggling percentages, the way risk researchers recommend in a review of how doctors and patients read health statistics. Say 99 percent accurate means two things: the test catches 99 percent of people who have the disease, and it wrongly flags 1 percent of people who do not. Now picture 100,000 people and a disease that affects 1 in 1,000 of them.
Sick people: 1 in 1,000 of 100,000 is 100.
True positives: the test catches 99 of those 100.
Healthy people: 100,000 minus 100 is 99,900.
False positives: 1 percent of 99,900 is 999.
Everyone who tests positive: 99 plus 999 is 1,098.
Chance a positive is real: 99 out of 1,098, about 9 percent.
So roughly 10 of every 11 positive results are false alarms. The test is not bad; the disease is rare, so flagged healthy people outnumber the sick ten to one.
Now keep the same test and change only the base rate:
Disease in 1 in 1,000 people: a positive is real about 9 percent of the time.
Disease in 1 in 100 people: 990 true positives against 990 false ones, so 50 percent.
Disease in 1 in 10 people: 9,900 true positives against 900 false ones, about 92 percent.
Do doctors fall for the base rate fallacy?
Often, yes. In a 1978 study in the New England Journal of Medicine, Casscells and colleagues asked 60 doctors, house officers and medical students about a disease with a prevalence of 1 in 1,000 and a test with a 5 percent false positive rate. According to a 2014 replication in JAMA Internal Medicine, only 11 of the 60 gave the correct answer of about 2 percent, and the most common answer was 95 percent.
The replication put the same question to 61 doctors and students at a Boston-area hospital in 2013 (full text on PubMed Central). Only 14 of 61, or 23 percent, got it right. Another 27 answered 95 percent, and the median answer was 66 percent, about 33 times the true value.
The 2 percent is the same counting, assuming the test catches every true case: per 1,000 people, 1 true positive against about 50 false ones.
How do you avoid the base rate fallacy?
The health statistics review by Gerd Gigerenzer and colleagues recommends natural frequencies instead of conditional probabilities, which is what the count above does. A four-step routine for any problem of this kind:
Pick a big round crowd, such as 100,000 people.
Split it by the base rate first: how many have the condition, how many do not.
Apply the hit rate to the first group and the false alarm rate to the second.
Divide true positives by all positives.
This routine is Bayes' theorem written as counting. We work through the formula version in our Bayes' theorem post with a 10,000-person count. The same split shows up in two-way table conditional probability problems, and if you think better in branches, build a conditional probability tree diagram and simulate it.
When a probability problem goes wrong and you cannot see where, the Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer.
What does a positive test result really tell you?
The chance that a positive result is real is called the positive predictive value, the quantity the JAMA Internal Medicine paper asked about. A positive screening result is a probability to interpret, not a diagnosis. Three questions worth asking:
How common is this condition in people like me?
If this test is positive, how likely is it that I really have it?
What is the follow-up test, and how is it different?
A follow-up test helps because the first positive raises the starting probability: in our example, from 1 in 1,000 to about 9 percent. If a second, independent test with the same accuracy is also positive, the chance rises to about 91 percent. Real follow-up tests usually work differently and are not fully independent, so this sum is an illustration, not a reading of any real result.
Next time a number sounds certain, ask for the base rate first, then count.
