On this page
"Should my child use a learning app?" is the wrong question. A study from Gujarat suggests a better one: what will the app replace? The same maths software helped children in one setting and made them learn less in another. This post explains the study and what it means for how students use technology today.
What question did the study ask?
The paper is Complement or Substitute? The Effect of Technology on Student Achievement in India by Leigh Linden of Columbia University and J-PAL, a working paper dated June 2008. It evaluated a computer-assisted learning (CAL) programme in schools run by Gyan Shala, an NGO in Gujarat, and asked whether the effect depended on how the computers were fitted into the school day.
What were the schools and the software like?
According to the paper, Gyan Shala ran low-cost schools for children in grades 1 to 3 in Ahmedabad's urban slums and in villages, with women from the local community as teachers and a tightly scripted curriculum. The paper notes that these were relatively well-functioning schools.
The CAL software was designed to reinforce that day's maths lesson, not to let children race ahead. It ran on old donated desktops, with two children sharing a split screen, one using the keyboard and one the mouse. Teachers only had to switch the machines on. The target cost was about five dollars per child per year, not counting power.
How was it tested?
Schools were randomly assigned to get the programme or to continue with the normal curriculum, in two separate years.
2004-05, in-school model (Patdi and Dhrangadhra). Children used the computers during the regular three-hour school day, in place of part of the normal lesson. 11 schools with 392 students got the programme; 12 schools with 387 students were the comparison group.
2005-06, out-of-school model (Ahmedabad and Halol). Children used the computers before or after their class, so it added to the normal lesson. 19 schools with 682 students got the programme; 18 schools with 695 students were the comparison group.
The paper notes that Gyan Shala had planned to run the out-of-school model in both years, but local administrators in the first area chose the in-school model instead.
What did it find?
The abstract gives the main results:
As a substitute for class time, the in-school model caused students to learn significantly less than they otherwise would have: 0.57 standard deviations lower.
As a complement, the out-of-school model raised average maths scores by 0.28 standard deviations. Most students had small gains that were not statistically significant on their own, while the weakest and older students gained 0.4 to 0.69 standard deviations.
The author's explanation is that the software was "a poor substitute for the teacher delivered curriculum". In a school where teaching was already good, taking time away from the teacher to use the computer was a loss.
What does the study not show?
The two models were tested in different years and places. The comparison between in-school and out-of-school use is not from a single randomised contest between the two, so differences between areas could play a part.
It is a working paper on young children. The students were in grades 2 and 3, and the software was simple drill software from the mid-2000s.
The schools were already good. In a weaker school, the same software might replace less useful time. The paper itself suggests this could explain why CAL has done better in some less well-functioning systems, such as the Vadodara schools where an earlier trial by Banerjee, Cole, Duflo and Linden found that computer maths games raised scores.
What does it mean for students today?
The lesson applies directly to AI tools and apps. If an AI tool replaces your own thinking during homework, or replaces time you would have spent with a good teacher, it can make you learn less. If it is added on top, for extra practice, feedback or a second explanation after you have tried, it is more likely to help.
A practical test: before opening an app, ask "what would I be doing otherwise?" If the answer is "solving this problem myself", try first, then use the tool to check. Our post on cognitive offloading and AI covers newer research on the same risk. For a study where technology was added as after-school practice and worked, see the Mindspark study in Delhi.
How to read a study like this
When a study reports an average effect, look for the breakdown. Here, the overall average hid one positive result and one negative result, and within the positive result most of the gain went to weaker and older students. An average of "small effect" can hide a story that matters for you.
When you use a step-by-step solver, use it after your own attempt: the Fermi Problem Library explains the reasoning behind each step, so it works as a complement to your thinking rather than a replacement.
Tonight, attempt each homework problem for ten minutes on your own before you open any app.
