On this page
In autumn 2023, physicists at Harvard ran a trial in Physical Sciences 2, an introductory physics course for life-sciences students. Students were randomly split into two groups: one learned a topic from an AI tutor at home while the other learned it in an active learning class; the next week they swapped. The results were posted as a preprint in May 2024 (Research Square) and published in June 2025 in Scientific Reports (Kestin et al., 2025).
What did the Harvard AI tutor study find?
Figures here come from the open-access paper (full text on PubMed Central).
Scores: the median post-test score was 4.5 after the AI lesson and 3.5 after the in-class lesson, from a pre-test median of 2.75.
Gains: measured from that shared 2.75 starting point, the AI group's median gain (1.75 points) was more than double the in-class gain (0.75 points).
Time: the median AI student spent 49 minutes, against about 60 minutes of learning time the authors assume for the 75-minute class.
Feelings: on a five-point scale, engagement averaged 4.1 after the AI lesson and 3.6 after class; motivation averaged 3.4 and 3.1. Enjoyment and growth mindset showed no significant difference.
The authors estimate the effect at 0.73 to 1.3 standard deviations, which they describe as large.
How was the study designed?
The comparison was not AI against a dull lecture. The control was the course's usual active learning class: students worked through physics activities in self-selected groups, with help from course staff and feedback from the instructor (Kestin et al.). For the evidence behind that style, see what the 2014 Freeman active learning meta-analysis found.
The tutor ran on GPT-4, but it was not GPT-4 left to itself. The team:
wrote a system prompt asking it to keep students active, manage cognitive load and encourage a growth mindset;
split each lesson into a sequence of problems, so students moved one step at a time;
put instructor-written, step-by-step solutions into the prompts, so it did not have to work out answers itself;
let students go at their own pace, with pre-recorded videos.
The topics were surface tension in week one and fluid flow in week two, each with a pre-test and a post-test. Of 233 students enrolled, 194 met the eligibility rules. Every student had both conditions: a crossover randomised controlled trial (paper).
What does the Harvard study not show?
The authors flag most of the limits themselves (Kestin et al.).
Long-term learning. The post-test came straight after each lesson. No delayed test was reported.
That any chatbot works. The authors say the gains may depend on "expert-crafted, question-specific prompts written by instructors experienced with the content" and a carefully structured framework.
Harder thinking. They "do not presume that structured AI tutoring will always outperform in-class active learning in all contexts", such as tasks that combine many concepts.
Replacing class. They advise against AI entirely replacing in-class teaching, and suggest using it to teach introductory material before class, freeing class time for higher-order skills.
"Over double" depends on the maths. Clay Ford of the University of Virginia Library StatLab re-analysed the data the authors shared (re-analysis, January 2026). Comparing each student with their own pre-test rather than the overall median, he estimated the AI group's post-test scores at about 12% to 26% higher (a 95% confidence interval). "Yes, absolutely, the AI group seems to do better. But I wouldn't say it doubled the effect," he wrote.
How can you use these findings in your own physics study?
The parts that made the tutor work can be copied at home. Here is a 50-minute session built on the study's design, with Class 11 surface tension as the example.
Pre-test (5 minutes). Answer three questions cold: why a small water drop is nearly spherical, why the excess pressure inside a soap bubble is 4T/r and not 2T/r, and what decides how high water rises in a thin tube.
Introduction (10 minutes). Read one textbook section or watch one short video.
Guided problems (25 minutes). Work four or five problems, easiest first. Try each yourself, ask for a hint rather than the answer, and check your method line by line against a worked solution.
Post-test (10 minutes). Answer three new questions of the same type and compare them with your pre-test.
Take whatever you still cannot do to your teacher or a study group. That mirrors the authors' own suggestion: AI for first contact, people for the harder problems.
Before relying on any AI tutor, run the checks in how to evaluate an AI tutor in 20 minutes, and read what tutoring research says about hints versus answers.
We built Fermi's AI tutor to teach rather than answer: it works out what you already know, builds a plan for the session and checks understanding as it goes. If your method keeps going wrong in step 3, the Fermi Problem Library has a step-by-step solver that explains the reasoning rather than printing an answer.
Try the plan on one topic this week and compare your two tests.
