On this page
Does practising maths with a chatbot help you learn? A June 2025 field experiment in the Proceedings of the National Academy of Sciences (PNAS) tested this with nearly 1,000 high-school students, and the answer depended on how the AI was set up.
What did the Bastani 2025 study test?
Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman, mostly from the University of Pennsylvania, ran a randomised controlled trial at one large high school in Turkey in the autumn semester of the 2023-24 school year (paper in PNAS). According to the full text, they ran four 90-minute maths review sessions for about fifty Class 9, 10 and 11 sections, covering roughly 15% of each grade's semester syllabus.
In each session the teacher revised a topic, students solved practice problems, then sat a closed-book exam in which each question matched a conceptually similar practice problem. Whole classrooms were randomly assigned to one of three arms for the practice part only:
Control: textbook and notes, no devices.
GPT Base: a GPT-4 chat window similar to ChatGPT, prompted only to act as a tutor for the problem.
GPT Tutor: the same chat window, but its hidden prompt told it to give hints and not the answer, and it held teacher-written solutions and common student mistakes.
Both the practice and the exam counted towards students' final grades.
Does using ChatGPT for practice hurt exam scores?
In this study, yes, when the AI had no guardrails. The paper reports that during practice, students with GPT Base scored 48% higher than the control group and students with GPT Tutor scored 127% higher.
Students who had practised with GPT Base scored 17% lower than the control group once the laptops were closed, a statistically significant drop (0.054 on a 0-1 scale against a control average of 0.321) (Table 1). Students who had used GPT Tutor scored statistically the same as the control group: the harm largely disappeared, but there was no measurable gain either.
Why did students do worse?
One explanation the authors tested was that the AI misled students: asked "What is the answer?" ten times for each of 57 practice problems, GPT Base was right only 51% of the time, with logical errors 42% of the time (full text). Those errors lowered practice scores, but the authors found no evidence that they carried over to the matching exam problems.
The better-supported explanation was the "crutch". Students often asked GPT Base for the answer and copied it. Looking at students' first message on each problem in session one, 67% of GPT Base messages simply restated the question or asked for the answer, against 37% for GPT Tutor (full text). GPT Base students did not think they had learned less, and GPT Tutor students overrated how well they did, which matches the gap between feeling and learning described in the 2019 physics study on the feeling of learning.
What does the study not show?
The authors list the limits themselves (PNAS):
One school, one subject (maths), one country.
Short-term learning only: an exam at the end of the same session, not marks months later.
It ran in autumn 2023 with GPT-4; models and student habits have changed since.
The guardrailed tutor stopped the harm but did not raise exam scores.
Other trials point the other way under different conditions. A 2025 Scientific Reports trial with Harvard physics students found they learned more in less time with a research-based AI tutor than in an active-learning class (Kestin et al., 2025); we cover it in our explainer on the Harvard AI tutor physics study. Together, the two suggest that how the tool is designed and used matters more than whether AI is present.
A separate 2024 PLOS ONE study compared ChatGPT-written maths hints with human tutor hints, not open chat access against no AI, so it does not test the crutch effect; see our explainer on ChatGPT hints vs human tutor hints.
What should Indian students take from this?
What follows is our inference, not a finding of the study. The students did Class 9 to 11 school maths, close to boards and JEE Main work, but Turkish classrooms are not Indian schools or coaching batches.
A simple rule: use AI to get unstuck, never to get finished. Try this four-step routine for any problem set:
Attempt each problem alone for at least 10 minutes and write down where you got stuck.
Ask the AI for one hint about that exact step, not the solution. A useful script: "Don't solve it. Tell me only what I should check next in my working."
Finish the problem yourself, then ask the AI to check your final step.
Next day, redo two of those problems with no AI and no notes. If you cannot, you were copying, not learning.
The same test applies to any AI tool you use, including Fermi's AI tutor: can you solve the next problem with the screen off? Parents can ask: "Show me one you can do without help."



