On this page
An AI tutor can look impressive in a demo and still be bad for learning. Some of the clearest evidence comes from a 2025 field experiment in PNAS by Bastani and colleagues with nearly a thousand high school maths students. Students with a plain ChatGPT-style tool got 48% higher grades on practice problems, but once it was taken away they scored 17% lower than students who never had it. A second version with prompts designed to protect learning largely avoided that harm. Same underlying model, different design, very different results.
So the question is not "is this AI clever?" but "does this AI make me do the thinking?" The eight tests below answer that in about 20 minutes. You need one problem you have already solved correctly, one you got wrong, and a timer. For background on why design matters, see our piece on AI tutors vs answer chatbots.
What should a good AI tutor do?
Researchers have started to spell this out. In a 2025 randomised trial at Harvard, Kestin and colleagues designed a physics tutor around seven research-based practices, including "facilitating active learning", "managing cognitive load", "scaffolding content", "ensuring accuracy of information and feedback" and "allowing for self-pacing". The same paper names a well-known flaw of AI tutors: their "uncanny confidence when giving an incorrect answer or when marking a correct reply as incorrect."
Older research points the same way. Kurt VanLehn's 2011 review in Educational Psychologist sorted computer tutors by whether they respond only to final answers or to each step of a solution. Intelligent tutoring systems, most of which work step by step, reached an effect size of 0.76, close to human tutoring at 0.79. A good tutor engages with your steps, not just your answer.
The 8 tests (about 20 minutes)
The "just tell me" test (2 min). Paste a homework problem and write "just give me the answer". A tutor built for learning asks what you have tried or offers a first hint. One that prints a full solution straight away is an answer engine.
The planted-error test (3 min). Type out a solution with one deliberate mistake, such as a sign error in step 3. Does it find the exact step and explain why it is wrong, or only say the final answer is off?
The unusual-method test (3 min). Submit a correct solution that uses a different method from the textbook, such as energy conservation instead of kinematics. A weak tutor marks it wrong because it does not match its own route.
The known-answer test (3 min). Give it a hard problem whose answer you already know. Check the final value and the units. On JEEBench, 515 problems from JEE Advanced, the best model tested in 2023 (GPT-4) scored under 40%. Models change fast, so treat that as a snapshot, and check anyway.
The "still confused" test (2 min). After an explanation, reply "I still don't get it" twice. Good tutors change the representation: a diagram, a simpler case, a question. Weak ones repeat themselves.
The check-back test (2 min). Does the tutor ever ask you a question to confirm you understood, or does it assume you did?
The memory test (2 min). Later in the session, refer to the mistake from test 2: "Is this the same error as before?" A tutor that holds the thread of the session will connect them.
The missing-data test (3 min). Give a problem with a needed value left out. A good tutor asks for it or states an assumption clearly. A weak one quietly invents a number.
How do you score the results?
Score each test 0 (failed), 1 (partly) or 2 (passed), out of 16. This is our rough guide, not a research cut-off:
13 to 16: good for daily practice, with your normal checks on final answers.
8 to 12: usable for some jobs, such as quizzing you, but not as your main explainer.
Below 8: treat it as a search tool, not a tutor.
Two results matter more than the total. Failing test 1 means the tool makes copying easy, which is the pattern the Bastani study warns about. Failing test 3 or 4 means you cannot trust its marking without checking, which our guide on checking an AI's physics solution covers step by step.
What should parents look for?
Sit with your child for the 20 minutes. Watch who does more of the typing and thinking. If the AI writes long solutions while your child reads, that is the crutch pattern. If your child is answering questions and fixing steps, that is tutoring. Our parent's guide to children and ChatGPT for homework has more on setting rules at home.
Run the tests on us too
We built Fermi's AI tutor to teach rather than answer: it holds a line of reasoning across a session, notices where you are stuck, and decides whether to explain, ask or show. Please hold it to the same eight tests. Any tutor worth using should be able to face them in front of you.
