On this page
AI tools write physics solutions that look finished: clean steps, tidy formulas, a boxed answer. That polish is the problem. A wrong solution looks just as confident as a right one, so you need your own way to check.
How often do AI models get physics wrong?
It depends on the model and the problem, but the research shows it is common enough to matter. When Arora, Singh and Mausam built JEEBench, 515 maths, physics and chemistry problems from JEE Advanced, the best result from any model they tested in 2023 was under 40%. The typical failures of GPT-4, the best model, were "errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts". Models change fast, so treat that score as a snapshot, but those three failure types are a useful list of what to look for.
In a 2023 study in Physical Review Physics Education Research, Gerd Kortemeyer graded ChatGPT on the assessments of a real calculus-based physics course. It would "narrowly pass" while "exhibiting many of the preconceptions and errors of a beginning learner". He also noted that language models' "convincingly presented arguments and facts are frequently simply false."
Why do AI models sound sure when they are wrong?
A 2025 paper by Kalai and colleagues compares it to "students facing hard exam questions": models "sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty." They argue that the way models are trained and evaluated rewards guessing over admitting uncertainty. In other words, a confident wrong number often scores better than "I'm not sure". Researchers at Harvard who built a physics AI tutor list "uncanny confidence when giving an incorrect answer" as a well-known flaw of AI tutors in their 2025 trial, and the accuracy of their own tutor relied on pre-written answers.
A five-check routine for any AI solution
Run these in order. Most take under a minute.
Units. Do both sides of the final formula have the same dimensions? Is the answer in the right unit? (Our post on dimensional analysis covers this in detail.)
Limiting cases. Set an angle to 0° or 90°, a mass to zero or very large, a distance to infinity. Does the formula give what common sense says?
Signs and directions. Is the chosen positive direction stated, and is it used the same way in every step?
Size. Is the number reasonable? A ball thrown by hand rising 2 km is wrong, whatever the algebra says.
Re-derive one key step. Pick the step that carries the physics, such as time of flight or a force balance, and do it yourself.
Worked examples: what each check catches
Example 1: a pendulum. Suppose an AI writes the period of a simple pendulum as T = 2π√(g/L). Check units: g/L has units of (m/s²)/m = 1/s², so √(g/L) is in 1/s. A period cannot be measured in 1/s. The correct formula is T = 2π√(L/g), which gives seconds. The units check catches this in ten seconds.
Example 2: projectile range. Suppose the AI gives the range on level ground as R = u² sin θ cos θ / g. Units are fine: (m²/s²)/(m/s²) = m. Limiting cases also pass: at θ = 0 and θ = 90° the range is zero, as it should be, and the maximum is at 45°. Yet the formula is wrong by a factor of 2.
Only check 5 catches it. Time of flight is T = 2u sin θ / g. Horizontal speed is u cos θ. So R = u cos θ × 2u sin θ / g = 2u² sin θ cos θ / g = u² sin 2θ / g. The missing 2 came from the time of flight.
What to do when you catch a mistake
Tell the AI the specific check that failed. "Your formula gives a period in 1/s" is more useful than "that's wrong".
Do not trust the corrected answer automatically. Run the five checks again.
Log it. Write the error type in your error log: algebra slip, wrong concept, sign, missing factor. Over a month you will see which checks you need most. Our post on why JEE scores plateau explains how an error log helps.
Use it as practice. Spotting an error in someone else's solution is good training for spotting your own in the exam hall.
The Fermi Problem Library has worked problems with a step-by-step solver that explains the reasoning rather than printing an answer, so you can compare each line with your own. For a fuller way to judge any AI tool, try our 8 tests for evaluating an AI tutor.
