On this page
AI tutors feel new, but computer tutors that follow a student's steps and give feedback have been built and tested for decades. That research tells us a lot about what computer tutoring can and cannot do, and it is more useful than many of today's headlines. Here is what the main studies found, in order, and what the findings mean when you choose or use an AI tutor now.
What were intelligent tutoring systems?
An intelligent tutoring system (ITS) is a program that models what a learner knows and gives individual instruction as they work. One well-known family was the Cognitive Tutors. A 1995 review by Anderson and colleagues looked back on ten years of building them for LISP programming, geometry and algebra (Anderson, Corbett, Koedinger and Pelletier, 1995).
Three findings from that review still matter:
In the best cases, students reached at least the same level of proficiency as with conventional instruction in about one third of the time. Early evaluations "usually, but not always" showed significant gains.
The most effective style was immediate feedback with short, directed error messages.
The tutors seemed to work better presented as non-human tools to help learning than as imitations of human tutors.
How did computer tutors compare with human tutors?
A widely cited answer is Kurt VanLehn's 2011 review in Educational Psychologist (VanLehn, 2011). He sorted computer tutors by how finely they interact: answer-based systems check only your final answer, step-based systems respond to each step of your working, and substep-based systems go finer still.
A popular belief held that human tutors were about two standard deviations better than no tutoring, the idea behind Bloom's "2 sigma problem", which we cover in Bloom's 2 sigma problem. VanLehn's review did not confirm that. He found an effect size of about 0.79 for human tutoring and 0.76 for intelligent tutoring systems: smaller than the legend, and nearly equal to each other.
Two meta-analyses broadly agreed:
Ma, Adesope, Nesbit and Liu pooled 107 effect sizes from 14,321 participants (Journal of Educational Psychology, 2014). ITS beat teacher-led large-group instruction (g = 0.42), other computer-based instruction (0.57) and textbooks or workbooks (0.35). They found no significant difference between ITS and individual human tutoring, or small-group instruction.
Kulik and Fletcher analysed 50 controlled evaluations and found a median gain of 0.66 standard deviations, roughly from the 50th to the 75th percentile (Review of Educational Research, 2016). But results depended heavily on the test: gains were larger on tests written locally to match the tutor than on standardised tests.
What happened when tutors were used at scale?
A large trial put Cognitive Tutor Algebra I into middle and high schools across seven states, randomly assigning matched pairs of schools (Pane and colleagues, 2014). In the first year there was no effect. In the second year there was evidence of a positive effect, statistically significant in high schools, worth about eight percentile points for the median student. Schools needed time to learn how to use it.
What does this teach us about AI tutors today?
Today's chatbots can talk about anything, which older systems could not. But the old findings give a useful checklist, whether you are a student or a parent.
Does it respond to your steps, not just your answer? Step-level feedback is where VanLehn found the gains. A tool that only says right or wrong is an answer-based system, however friendly it sounds.
Is its feedback short and specific? The Cognitive Tutor work found short, directed messages worked best. Long essays in reply to a small error are easy to skim and forget.
Does it make you do the work? All of these systems had students solve problems while the tutor watched and responded. Help that solves the problem for you skips the part that produced the learning.
Is it measured fairly? Ask whether claimed results come from standardised tests or the tool's own quizzes, since Kulik and Fletcher found that difference matters.
Give it time. As in the Cognitive Tutor Algebra trial, a new tool takes a while to fit into a routine. Judge it after a few weeks of steady use, not one evening.
Older systems also tracked which skills a student had mastered before moving on; our explainer on mastery learning and adaptive practice covers that idea, and how to evaluate an AI tutor turns this checklist into tests you can run.
Where Fermi fits
We designed Fermi's AI tutor to teach rather than answer: it holds a line of reasoning across a whole session, notices where a student is stuck and decides whether to explain, ask or show. Whatever tool you use, the decades of research above give you the questions to ask of it.
