On this page
You paste a physics question into a chatbot, get 50 km/h, ask again and get 48 km/h. You did not imagine it. It comes from how these systems write, and once you know why, the disagreement becomes useful: it points to exactly the line you need to check.
Why does AI give different answers to the same question?
Three things change the answer even when your question looks the same.
Sampling. A chatbot writes its reply one small piece (a token) at a time, picking each piece from a list of likely options. Temperature is the setting that controls how random that pick is, and Anthropic's glossary says lower temperatures give "more conservative and deterministic outputs" while higher ones give more variety. The same glossary adds that even with temperature set to 0, "identical inputs might produce different outputs across API calls". In a 2025 test by Thinking Machines Lab, one prompt sent 1,000 times to an open model at temperature 0 produced 80 different completions.
Wording. Small changes in how a question is phrased or laid out can move the result a long way. A study presented at ICLR 2024 found performance differences of up to 76 accuracy points on one open model, LLaMA-2-13B, from prompt formatting changes alone in few-shot tests.
Context. The chatbot reads everything earlier in the conversation, plus any hidden instructions from the app. A different chat history, a different app or a different model is a different starting point, so the same final question can land differently.
Does disagreement mean the AI is wrong?
Not always, but it is a strong hint to check. The SelfCheckGPT study (EMNLP 2023) rests on a simple idea: if a model knows something, its repeated answers tend to agree, and for made-up facts, repeated answers "are likely to diverge and contradict one another". A 2024 paper in Nature by Sebastian Farquhar and colleagues built on this, measuring how much a model's answers vary in meaning to flag "confabulations", which it defines as arbitrary and incorrect generations.
The reverse also works. Sampling several lines of reasoning and taking the most common answer, a method called self-consistency, beat a single chain of reasoning on the GSM8K set of maths word problems; the paper's abstract reports a gain of +17.9% there.
A physics example: is the average speed 48 or 50 km/h?
Here is a classic question that can produce two different answers: "A car covers the first half of a trip at 40 km/h and the second half at 60 km/h. Find its average speed." Answer A says 50 km/h. Answer B says 48 km/h. Instead of picking the one that sounds more confident, check it.
Go back to the definition. Average speed is total distance divided by total time. If each half is a distance d, the time is d/40 + d/60 = d/24 hours. Total distance 2d divided by d/24 gives 48 km/h.
Find the assumption. If "half" means half the time instead, the car covers 40t + 60t = 100t in time 2t, which is 50 km/h. Answer A was not careless arithmetic; it answered a different reading of the question.
Try an extreme case. Imagine the second half at almost 0 km/h. Over a fixed distance the trip takes almost forever, so the average must fall towards 0. The half-distance formula, 2 × 40 × v ÷ (40 + v), does that. The simple average (40 + v) ÷ 2 stays near 20 km/h, which only makes sense for half the time.
In a textbook, "half the trip" usually means half the distance, so 48 km/h is the expected answer. The disagreement did its job: it showed you the one word that decides the answer.
How do you check when two AI answers disagree?
Use this five-minute routine.
Line the two solutions up and find the first step where they differ. Everything before it is probably shared.
Ask each one: "Which assumption did you make at this step? Quote the words in the question that support it."
Run your own checks: the definition, a dimensions check, an extreme case and a rough estimate before trusting the number.
Rewrite the question so only one reading is possible, then ask again in a fresh chat.
If it still splits, compare with your textbook or teacher. Our five-step check for when the answer key and AI disagree covers that case.
If the split is about method, the Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer, so you can see which step decides the result. And Fermi's AI tutor holds a line of reasoning across a whole session and notices where a student is stuck, so a question like "why 48 and not 50?" becomes something you work through together.
Next time two answers disagree, do not just pick one. Find the step where they split and check it.
