On this page
When you paste a tricky maths or physics problem into a conversational chatbot, the answer usually arrives in an authoritative, polished tone. Even when the reasoning skips a boundary condition or invents an algebraic shortcut, the model rarely hesitates. It states erroneous derivations with the exact same poise as basic arithmetic.
Why does this happen? Large language models do not possess an inner sense of doubt. They generate text by predicting statistically probable sequences. Furthermore, modern chatbots are fine-tuned using reinforcement learning from human feedback, which inadvertently penalizes hesitation. In an empirical study presented at ICLR 2024, Miao Xiong and co-authors evaluated confidence elicitation across 5 large language models and 5 reasoning benchmarks. They found that models systematically exhibit poor calibration and extreme overconfidence when asked directly how confident they are in an answer. The models mimic human conversational assertiveness rather than calculating true epistemic probability. Simply asking "Are you sure?" almost always prompts a cheerful confirmation followed by a restatement of the same blunder.
This confidence trap explains why AI chatbots get maths wrong. To make an AI say when it is unsure, you cannot ask for an overall confidence rating. You have to force the model to inspect its reasoning step by step.
How do you make an AI say when it is unsure?
Language models become far more calibrated when evaluating narrow, concrete statements rather than broad impressions. Research by Saurav Kadavath and colleagues showed that models evaluate whether a specific intermediate claim is valid far more reliably than they evaluate overall knowledge.
To elicit uncertainty effectively, use a prompt that mandates step-by-step classification:
"Solve the following problem step by step. For every intermediate step, provide:
The mathematical theorem, definition, or physical law applied.
An explicit uncertainty tag:
[CERTAIN]: Standard arithmetic, direct definitions, or reversible algebraic identities with no edge conditions.
[NEEDS CHECK]: Any operation that changes roots (such as squaring both sides or dividing by an expression), sign conventions, domain boundaries, or idealised physical assumptions.
At the end, compile a 'Step Audit List' containing only the steps tagged [NEEDS CHECK], stating exactly what constraint must be checked."
A worked example: spotting hidden algebraic traps
Consider what happens when you solve the equation √(2x + 3) = x for real values of x.
Under a default prompt, chatbots frequently square both sides to get 2x + 3 = x², rearrange into x² - 2x - 3 = 0, factor into (x - 3)(x + 1) = 0, and announce two solutions: x = 3 and x = -1. But plugging x = -1 back into the original equation produces √1 = -1, which is false for the real principal square root. The model reports a false solution because it overlooked domain constraints.
With the step-auditing prompt, the AI outputs an audit trail:
Step 1: Write equation √(2x + 3) = x. [CERTAIN]
Step 2: Establish the radical domain: 2x + 3 ≥ 0, so x ≥ -1.5. [CERTAIN]
Step 3: Square both sides to obtain 2x + 3 = x². [NEEDS CHECK: Squaring both sides is irreversible unless both sides share the same sign. This requires x ≥ 0 and may introduce extraneous roots.]
Step 4: Rearrange into standard quadratic form: x² - 2x - 3 = 0. [CERTAIN]
Step 5: Factor into (x - 3)(x + 1) = 0, yielding tentative roots x = 3 and x = -1. [CERTAIN]
Step 6: Step Audit List highlights Step 3: verify candidates against x ≥ 0. Testing x = -1 yields √1 ≠ -1, so discard x = -1. Final valid solution: x = 3.
The model flags the precise mathematical branch where extraneous roots enter.
How do you audit an AI uncertainty checklist?
When the AI produces its Step Audit List, treat it as a targeted verification plan. Never ask the AI to re-verify its own work; perform these three checks yourself:
Boundary and domain tests. Check whether an algebraic step assumed positive values or risked dividing by zero. In trigonometry, confirm angles do not cross asymptotes.
Sign conventions and coordinate frames. In mechanics, steps tagged with [NEEDS CHECK] often conceal direction mix-ups. We have documented when AI gets physics wrong: models frequently jumble coordinate signs while presenting derivations with total grammatical confidence.
Limiting cases. Substitute zero, unity, or extreme values into both sides of the flagged step to see whether the physical or mathematical behavior remains sound.
Why does step-by-step verification beat answer checking?
Step-level auditing mirrors how AI reasoning research evaluates complex problem-solving. In a study on mathematical reasoning, Hunter Lightman and colleagues evaluated process supervision (evaluating every intermediate step) against outcome supervision (evaluating only the final result). Using the PRM800K dataset containing 800,000 human step-level feedback annotations, their process-supervised model solved 78% of problems on a representative test subset of the challenging MATH benchmark. Catching errors early prevented faulty assumptions from compounding into confident, erroneous outcomes.
This distinction highlights the gap between AI tutors versus answer chatbots. General chatbots rush to output answers, whereas effective learning requires understanding where reasoning can fail.
Fermi's AI tutor works out what a student knows, notices where they are stuck, and decides whether to explain, ask, or show rather than simply generating an answer. For step-level practice, the Fermi Problem Library provides worked problems with a step-by-step solver that explains the reasoning rather than printing an answer.
Prompting an AI to highlight where it is unsure transforms it from an untrustworthy answer machine into a useful study partner. You let the machine do the heavy drafting, while you retain the critical verification role.
