On this page
When a chatbot writes a practice question or a worked solution, that text can become training material for the next AI model. This is synthetic data. It is quick to make, but it carries a risk any student who has borrowed a senior's notes will recognise: mistakes get copied, and the careful exceptions get dropped.
What is synthetic data in AI training?
The US National Institute of Standards and Technology defines synthetic data generation as a process in which seed data are used to create artificial data that have some of the statistical characteristics of the seed data. For language models, the seed is often real human writing, and the generator is often another AI model.
Why bother? Real data is limited and slow to label by hand. We explained who does that labelling in who labels data for AI training, and how to try it in class. Synthetic data can fill gaps on demand.
It can work well. Microsoft researchers trained phi-1, a 1.3-billion-parameter coding model, on 6 billion tokens of filtered web text plus 1 billion tokens of textbooks and exercises written by GPT-3.5, and it scored 50.6% on the HumanEval coding test (Gunasekar et al., 2023).
Why can AI-made training data spread errors?
In 2024, Ilia Shumailov and colleagues published a study in Nature asking what happens when models learn from text that earlier models wrote (Shumailov et al., Nature, 2024). They found that indiscriminate use of model-generated content in training causes irreversible defects, in which the tails of the original data, the rare cases, disappear. They called this model collapse.
A main cause is sampling. A rare case may not show up in the sample a model learns from, so the next model never sees it. In early model collapse, the paper says, the model loses information about the tails; in late collapse it settles on something with little resemblance to the original.
The team fine-tuned Meta's OPT-125m model on the WikiText-2 dataset, then trained each new generation on text from the one before. A prompt about medieval church building came back, by generation 9, as a list of black-tailed, white-tailed and blue-tailed jackrabbits (full paper on PubMed Central).
Collapse is not inevitable. In the same paper, keeping 10% of the original data in each generation led to only minor loss of performance. A separate 2024 study by Matthias Gerstgrasser and colleagues found that adding each generation's synthetic data on top of the real data, instead of replacing it, prevented collapse in their tests (Gerstgrasser et al., 2024).
What does model collapse look like in a student's notebook?
Here is a made-up but familiar chain of Class 11 maths notes, passed from senior to junior.
Generation 0: Aarav writes a margin note in class: the square root of x squared is the absolute value of x, not x. Check x = -3: the square root of 9 is 3, not -3.
Generation 1: Diya rewrites the notes neatly, keeping only the main points. The margin note looks minor, so she leaves it out.
Generation 2: Kabir copies Diya's notes. With no warning in front of him, he simplifies the square root of x squared to x in a worked example.
Generation 3: Four juniors learn from Kabir's version. Each now gets negative values wrong in the same way.
Nobody lied. Each copy was a little cleaner and a little less complete. That is model collapse in miniature: the rare case (a negative x) goes first, an error fills the gap, and the error becomes what everyone "knows". Kabir's tidy page is the jackrabbit list.
How do you check AI-made study material before you trust it?
Use this five-point check on any AI-written summary, question bank or solution:
Trace it to a source. Find each formula in your NCERT textbook or class notes before you copy it.
Test the edge cases. Plug in zero, a negative number or a very large value.
Look for missing exceptions. If the AI's version is neater than the chapter, ask what it left out.
Do not stack copies. Never make notes from an AI summary of another AI summary.
Ask for the reasoning, not just the answer. Step-by-step working shows where a slip entered.
For the last point, the Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer, so you can check each step against your textbook. Fermi's AI tutor checks understanding as it goes, which helps you catch a dropped exception before it becomes a habit.
Our guide on what to say when an AI tutor gives your child a wrong answer has a short script. To see how models are shaped after training, read fine-tuning vs prompting AI, explained for students.
Next time you borrow notes, AI-made or not, check one formula against the textbook before you trust the rest.
