On this page
An AI model learns by finding patterns in examples. If the examples are lopsided, the patterns are too. That is all "bias in training data" means: not a machine with opinions, but arithmetic done on a skewed sample of the world.
Below are three documented cases, each a different way data goes wrong, and a checklist for any AI tool.
What is bias in AI training data?
Bias in training data is a mismatch between the examples a model learns from and the people or situations it is later used on. It usually enters in one of three ways: some groups are missing or rare in the data, the label the model learns is a poor stand-in for what we actually care about, or the data comes mostly from one part of the world. People choose and label that data; our explainer on who labels data for AI training shows how that work is done.
Case 1: what happens when some people are missing from the data?
In 2018, Joy Buolamwini and Timnit Gebru published Gender Shades, a study of face-analysis systems. They first checked two face-analysis benchmarks and found them overwhelmingly made up of lighter-skinned subjects: 79.6% for IJB-A and 86.2% for Adience.
They then built a test set balanced by gender and skin type and evaluated three commercial gender classification systems on it. Darker-skinned women were the most misclassified group, with error rates of up to 34.7%, while the maximum error rate for lighter-skinned men was 0.8%, according to the paper's abstract.
The lesson: one overall accuracy figure can hide a group the tool fails. Always ask, "accurate for whom?"
Case 2: what if the AI learns the wrong target?
A 2019 study in Science by Ziad Obermeyer and colleagues examined a widely used US health-care algorithm that helps pick patients for extra care. The algorithm predicted future health care costs, not illness. Because unequal access to care means less money is spent on Black patients, Black patients given the same risk score as White patients were considerably sicker.
The authors estimated that remedying this would raise the share of Black patients receiving additional help from 17.7% to 46.5% (abstract via Europe PMC). The label, cost, was a convenient proxy that looked accurate by some measures yet carried an old inequality into a new system.
The lesson: ask what the tool was trained to predict, and whether that is the thing you care about.
Case 3: does it matter where the data comes from?
In 2017, Shreya Shankar and colleagues at Google Brain studied two large public image datasets. In their geo-located ImageNet sample, around 45% of images were from the US, while India had 2.1% and China 1%.
The effect showed up in a simple test. Photos of grooms supplied by crowd workers in Hyderabad were far less likely to be recognised correctly, and the ImageNet-trained model was likely to label them "chain mail", a kind of armour, the same paper reports. It studied image datasets, not chatbots, but the question carries over.
The lesson: a tool can be good in general and still weak on Indian names, places, syllabi and examples.
What questions should students ask of any AI tool?
Use these five before you trust an answer that matters, such as a career suggestion or a project summary.
Who is in the data, and who might be missing? (Case 1)
What was it trained to predict, and is that what I need? (Case 2)
Where is most of its data from, and does it handle Indian context: NCERT chapters, rupees, Indian names, state boards? (Case 3)
Does it work as well for people like me? Test it, as below.
Can I check the answer against something independent: the textbook, an official site, a teacher, or my own working?
How can you test an AI tool for bias at home?
This takes ten minutes. Pick a prompt where a person's background should not change the answer, then change only one detail.
Run A, five times: "Suggest three careers for Priya, a Class 11 student from Patna who loves physics."
Run B, five times: the same prompt with "Rahul" instead of "Priya".
Run C, five times: the same prompt with "Bengaluru" instead of "Patna".
Note each suggestion and compare. Are engineering and research suggested equally often? Do the reasons change? A pattern that repeats is a bias worth reporting to the app.
The same habit applies to science answers: ask for the reasoning, then check each step. The Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer, so every step is there to check.
Bias can also compound when models learn from text other models wrote; our guide to synthetic data in AI training explains how errors spread. And for quick answers at the top of a search page, see how reliable AI search summaries are.
Try the ten-minute test this week with a friend, and compare notebooks.
