On this page
You ask a chatbot a question and it answers in calm, tidy paragraphs. That smooth tone feels like knowledge, but it is not. Once you see where the confidence comes from, you can ask questions that expose the real uncertainty.
Why does AI sound so confident when it is wrong?
A large language model writes by producing likely next words, one after another (our explainer on how large language models work covers this). Fluent text is what it is built to make, so a wrong answer comes out just as polished as a right one.
The habit of guessing is also trained in. In a 2025 paper, Adam Tauman Kalai and three co-authors wrote that "like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty."
Their explanation will sound familiar to any JEE aspirant. Many tests used to score AI models give one point for a right answer and nothing for a blank or "I don't know". Under that rule, the paper notes, "guessing when unsure maximizes the expected score" (Kalai et al., 2025). The diagram above shows why: with no penalty for a wrong answer, a guess never scores below "I don't know". Add a penalty and guessing only pays when the model is fairly sure, which is why the authors suggest scoring tests that way.
In the paper's own example, asked for Kalai's birthday as DD-MM, "if you know", one model gave three different dates on three tries, all wrong.
Tuning a model for chat can make this worse. OpenAI's GPT-4 Technical Report says the pre-trained model was "highly calibrated", meaning its confidence matched how often it was right, but "after the post-training process, the calibration is reduced." The same report says GPT-4 "can also be confidently wrong in its predictions."
Why is tone not evidence?
People read fluency as accuracy. In a 2025 study in Nature Machine Intelligence, Mark Steyvers and colleagues found that users overestimated how accurate a language model's answers were when shown its standard explanations. Longer explanations raised people's confidence even when the extra length did not make the answers more accurate. When explanations were rewritten to reflect the model's own confidence, people's sense of how often it was right moved closer to the truth.
Can you just ask the AI how sure it is?
Partly. In a 2022 study, Saurav Kadavath and colleagues found that larger models were well calibrated on multiple-choice and true/false questions asked in the right format, and showed promise at judging whether their own answers were true. But a 2024 evaluation by Miao Xiong and colleagues found that models tend to be overconfident when they state their confidence in words, and every method they tested struggled on questions needing specialist knowledge. So the AI's own answer is a clue, not a verdict.
Four questions that reveal real uncertainty
"Which part of this are you least sure of?" Watch for named facts, dates and numbers it cannot point to.
"Answer it again from scratch." Better still, ask in a fresh chat. SelfCheckGPT, a method by Potsawee Manakul and colleagues, rests on the idea that when a model knows something its answers agree, while made-up details change from one try to the next.
"Where could I check this?" Then open the textbook, the NCERT chapter or the official page yourself. Links and references can be invented too, so read our guide on checking AI references and citations.
"Show me one step I can check myself." Units, a special case, putting the answer back into the question. Our five-step way to check AI maths goes further.
What does this look like on a real question?
Say you ask for the period of a simple pendulum 1 m long, and the chatbot replies, confidently, "about 1 second".
Question 4: ask for the formula. It gives T = 2π√(L/g). Put in L = 1 m and g = 9.8 m/s²: √(1/9.8) is about 0.32 s, and multiplying by 2π gives about 2.0 s.
Question 2: ask again in a new chat. If one chat says 1 s and another says 2 s, that disagreement is the uncertainty the tone was hiding.
Question 3: your physics textbook has the same formula, which settles it.
This is why working through a solution line by line beats reading a final number. The Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer, so each line is something you can test. Fermi's AI tutor checks understanding as it goes and decides whether to explain, ask or show. The four questions still apply to any AI, ours included.
What can parents say?
When your child quotes a chatbot, ask: "What did it say, and how did you check it?" Praise the check, not just the answer.
Next time an answer sounds very sure, ask one of the four questions before copying it down.
