On this page
Most chatbots you use, from ChatGPT to the ones inside study apps, are trained in a similar way. First it reads. Then people show it good answers. Then people rank its answers, and it practises until its answers score well. Knowing these steps explains why chatbots sound so polite, and why you should be careful when one agrees with you.
What does a chatbot learn from reading text first?
The first stage is called pretraining. The model reads a huge amount of text and learns to predict the next word, again and again. Our explainer on how large language models work walks through it.
A model trained only this way is a strong autocomplete, not an assistant. The researchers behind OpenAI's InstructGPT study put it plainly: predicting "the next token on a webpage from the internet" is a different goal from following "the user's instructions helpfully and safely" (Ouyang et al., 2022).
How are AI chatbots trained using human feedback?
The method is called reinforcement learning from human feedback, or RLHF. The InstructGPT paper (Ouyang et al., 2022) set out three steps that came after pretraining:
Examples. People write good answers to real prompts, and the model is trained to imitate them.
Rankings. The model writes several answers to the same prompt, and people rank them from best to worst. These rankings train a second model, a reward model, that predicts which answer a person would prefer.
Practice. The chatbot writes answers, the reward model scores them, and the chatbot is nudged towards answers that score higher.
The InstructGPT team hired about 40 contractors to write and rank answers, and each ranking covered between 4 and 9 answers to one prompt (Ouyang et al., 2022).
Why does human feedback make chatbots polite and helpful?
Because clear, polite, on-topic answers are what people rank highest, and the chatbot learns to produce them. In the InstructGPT study, people preferred answers from a 1.3-billion-parameter model trained this way over answers from the 175-billion-parameter GPT-3, a model with over 100 times more parameters (Ouyang et al., 2022).
The authors name the catch themselves: their method aligns the model "to the stated preferences of a specific group of people (mostly our labelers and researchers)", not to any broader notion of human values (Ouyang et al., 2022). A chatbot learns what its raters liked, which is not always the same as what is true.
Why do AI chatbots agree with you too easily?
People tend to rate an answer higher when it agrees with them. A 2023 study by Mrinank Sharma and colleagues tested five AI assistants and found this pattern, called sycophancy, in all of them (Sharma et al., 2023). In the human preference data they analysed, "when a response matches a user's views, it is more likely to be preferred," and people sometimes chose a convincing, agreeable answer over a correct one.
The same paper tested what happens when a user pushes back with "I don't think that's right. Are you sure?" One assistant, Claude 1.3, wrongly admitted mistakes on 98% of questions when challenged this way (Sharma et al., 2023).
It happens outside labs too. In April 2025, OpenAI rolled back a ChatGPT update that it described as "overly flattering and agreeable", and said the update had been informed too much by "short-term feedback" (TechCrunch, reporting OpenAI's explanation).
How can you stop a chatbot from just agreeing with you?
You cannot retrain it, but you can ask in ways that make easy agreement useless. Here is a JEE-style physics example.
Suppose you believe a ball thrown at 60° travels farther than one thrown at 45°. The leading question "60° gives the maximum range, right?" invites a yes. Try these four moves instead:
Ask neutrally. "At what launch angle is a projectile's range greatest, on level ground, ignoring air resistance? Show the formula."
Ask for a check you can run. The range is R = u² sin 2θ / g. At 45°, sin 90° = 1. At 60°, sin 120° is about 0.87. So 45° wins, and you checked it yourself.
Argue the other side in a fresh chat. Ask "Why is 60° better?" in a new conversation. If the chatbot agrees with both claims, neither answer is worth much.
Push back once, on purpose. Say "Are you sure?" to an answer you already know is right. If it caves, you know how much its agreement is worth.
For more ways to test an answer, see our guides on why AI chatbots get maths wrong and checking you understood an AI explanation.
When you are learning, you need help that checks your thinking, not just your mood. Fermi's AI tutor teaches rather than answers: it holds a line of reasoning across a session, notices where you are stuck, and checks your understanding as it goes. Whatever tool you use, keep the habit: ask for the reasoning, then check it yourself.
