On this page
Can an AI write the hints in a maths tutor as well as a human tutor can? A study from the University of California, Berkeley tested this directly. The headline sounds like good news for AI, and partly it is. But the details, especially what happened before learners saw any hints, matter just as much. This post walks through it.
What question did the study ask?
The paper is ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills by Zachary Pardos and Shreya Bhandari, published in PLOS ONE in May 2024. It is open access, so the full text can be read for free.
It asked two questions. How often are ChatGPT's worked-solution hints wrong? And do learners who see ChatGPT hints learn as much as those who see hints written by human tutors?
How were the hints made and checked?
The researchers used the "Dec 15 Version" of ChatGPT 3.5 to write one worked-solution hint for each problem, taken from open textbooks in four areas: elementary algebra, intermediate algebra, college algebra and statistics.
Before anyone saw them, every ChatGPT hint was checked for three things: correct final answer, correct working, and appropriate language. Any problem with a failed hint was removed from the study. On average, 32% of ChatGPT hints failed, from 25% in elementary algebra to 47% in intermediate algebra. Every failure was a wrong answer with wrong steps.
The team also tried "self-consistency", a technique from Wang and colleagues (2022): generating ten hints per problem and keeping the most common answer. That cut error rates to 0% for elementary algebra, 2% for intermediate and college algebra, and 13% for statistics.
How was the learning experiment run?
Participants were adults recruited online through Amazon Mechanical Turk, with at least a high school degree. Each person was randomly assigned to one of three conditions and one of the four subjects:
No hints (control): feedback on whether an answer was right, plus an option to see the answer.
Human tutor hints: worked solutions written by human tutors.
ChatGPT hints: the checked ChatGPT worked solutions.
Everyone did a three-question pre-test, five practice problems with their assigned help, and then the same three questions as a post-test. Of 394 people recruited, 274 completed the study: 90 in the control group, 86 with human hints and 98 with ChatGPT hints.
What did it find?
Learning gain was measured as post-test percent correct minus pre-test percent correct:
No hints: 1.85 percentage points, not statistically significant.
Human tutor hints: 11.62 percentage points.
ChatGPT hints: 17.00 percentage points.
Compared with the control group, only the ChatGPT condition showed a statistically significant extra gain (p = 0.011); the human-hint condition did not reach significance against control (p = 0.087). Crucially, the ChatGPT and human-hint gains were not statistically different from each other (p = 0.416). Time on task was also similar for the two hint groups.
What does the study not show?
Learners never saw the wrong hints. In everyday use, nobody filters ChatGPT's mistakes for you.
The participants were adults online, not school students. The median age group was 36-40.
The sessions were short. Three pre-test questions, five practice problems and the same three questions again tell us about short-term gains on very similar items.
It used an older model and text-only problems. Problems with figures were left out because the model could not handle images at the time.
What does this mean for Indian students?
Two takeaways. First, a good worked solution helps you learn, whoever writes it: both hint groups learned more than the no-hint group. Second, you cannot assume an AI's working is right. In this study, nearly one in three raw hints had wrong steps. When an AI explains a JEE or board-level problem, check the final answer against a trusted source and check whether each step follows. Our posts on why AI makes maths mistakes and hints or answers go further.
How to read a study like this
Read the methods before the headline. Here, the word "equivalent" depends on a filtering step that a quick reader might miss. Also check who the participants were and how long the learning lasted before applying the result to yourself.
When you want worked solutions you can follow step by step, the Fermi Problem Library has a step-by-step solver that explains the reasoning, so you can check each move rather than just the final answer.
Next time an AI solves a problem for you, cover the solution, redo it yourself, and compare step by step.
