On this page
A forward lands in the family WhatsApp group: "Scientists say this drink boosts memory." It may be true, half true, or a mouse study with a confident headline. Here is where hype usually comes from, and five questions to ask before you believe it or share it.
Where does science hype usually start?
Often not in the newsroom. A team led by researchers at Cardiff University compared 462 health-science press releases from 20 UK universities with the papers behind them and 668 news stories, in a study published in the BMJ in 2014.
They found that 40% of the press releases gave readers advice that went beyond the paper, 33% turned a correlation into a cause, and 36% stretched an animal result to people.
The news followed the press release. When a release overstated cause and effect, 81% of the news stories based on it did too; when the release stuck to the paper, only 18% did. For animal results stretched to humans, the figures were 86% and 10%.
So the headline is often two retellings away from the research. Get as close to the paper as you can, even if only its abstract.
What is a preprint, and should you trust it?
A preprint is a paper posted online before other scientists have reviewed it. It spreads results fast, but nobody outside the team has checked it yet.
The medRxiv server says it plainly: preprints "are preliminary reports of work that have not been certified by peer review" and "should not be reported in news media as established information."
If a story cites bioRxiv, medRxiv or arXiv, read it as "early and unchecked", not "proven".
Why do sample size and mice matter so much?
Small studies are noisy. Katherine Button and colleagues found that the average statistical power of neuroscience studies is very low, and that low power leads to overestimated effects and results that are hard to reproduce. Small samples are a main reason power is low.
Toss a fair coin 10 times and you will get 7 or more heads in about one set in six (176 of the 1,024 possible sequences). Get 700 heads in 1,000 tosses and something really is odd about the coin. A study of 20 students is the 10-toss version; our post on how margin of error shrinks as sample size grows shows the maths.
Animals raise a different problem: people are not large mice. A 2022 review in Acta Pharmaceutica Sinica B reports that nine out of ten drug candidates that enter clinical trials in people fail, and notes that biological differences between animal models and human disease make drug targets hard to validate. If "in mice" is in the paper but missing from the headline, treat that as a red flag.
Does "linked to" mean "causes"?
No. An observational study watches what people already do and looks for patterns. If students who sleep eight hours score higher, sleep might help, or calmer students might both sleep more and score better. A trial, where researchers assign the change and compare with a control group, is the most direct way to separate the two.
A third of the press releases in the Cardiff study made this jump themselves. For more, read our explainers on correlation versus causation in study news and how to spot a misleading graph.
The five-question check, with a worked example
Run any science headline through these:
Who is speaking? The paper itself, or a press release or news story about it?
Has it been peer reviewed? Or is it a preprint?
How many, and compared with what? A handful of people or thousands, and was there a control group?
In whom? Cells, mice, or people like you?
Did they change something, or only watch? A trial can suggest cause; an observation shows a link.
Practise on an invented headline: "Green tea makes teenagers better at maths, study finds." The abstract shows a survey of a few classes of students who reported their own tea habits and marks. It was peer reviewed and studied people (questions 2 and 4 pass). But nobody was given tea (question 5 fails), and a few classes of self-reported answers is a small, noisy sample (question 3 is shaky). A fair headline would be: "Students who drank green tea reported slightly higher maths marks in one small survey."
When someone forwards a claim like this, don't argue. Ask: "Interesting. Was that in people or in mice, and did they test it or just notice it?" For anything about your own health, take the question to a doctor, not a headline.
How can you get better at this?
The five questions rest on a few ideas from school science and statistics: control groups, sample size and probability. If any of those feel shaky, the Fermi Concept Library has a lesson for each concept in biology and mathematics. Fermi's AI tutor works out what you already know and decides whether to explain, ask or show, so you build the reasoning yourself.
Next time a headline surprises you, pick one question from the list and check it before you share.
