On this page
The hardest part of a multiple-choice question is not the right answer. It is the wrong ones. A good wrong option, or distractor, is picked by the student who holds a particular wrong idea, so the answer sheet tells you what they were thinking. AI can draft distractors in seconds; making them diagnostic takes your judgement.
Why are AI-written distractors often weak?
A team from the University of Massachusetts Amherst, the maths platform Eedi and two other institutions tested a range of large language models on about 1,400 real maths MCQs for students aged 10 to 13, published at NAACL 2024. Their conclusion: the models "can generate some mathematically valid distractors" but are "less adept at anticipating common errors or misconceptions among real students".
In the paper, plausibility measures how likely middle school students learning maths are to pick a distractor (full paper). Human raters scored AI-written distractors 3.28 out of 5 for validity and 2.68 for plausibility, against 3.99 and 3.72 for human-written ones. The gap was biggest for plausibility: the options were wrong, but not in the ways children are.
A smaller pilot by the same group, with four maths educators (two teachers, two tutors), found that 37% of AI-generated misconceptions, distractors and feedback were judged valid, against 70% of question stems, keys and explanations. The authors still saw promise in teachers and AI working together, which is the workflow below.
What makes a good MCQ distractor?
In a 1998 study in the Journal of Research in Science Teaching, Philip Sadler gave 1,250 students in grades 8 to 12 astronomy questions whose distractors matched common student ideas, at the start and end of their course. Such tests, he found, behave very differently from conventional ones and can track conceptual change. Teaching appeared to strengthen support for some wrong ideas before it declined, which suggests they may be steps towards understanding rather than obstacles.
So a good distractor has two jobs: be clearly wrong to anyone who understands the idea, and be the answer a student with a named misconception would choose.
How do you prompt AI to write distractors from misconceptions?
Give the AI the misconceptions first and ask it to work backwards. A prompt you can adapt:
"Here is my question and the correct answer: [paste]. My students are in Class [X], studying [chapter]."
"List the four most common misconceptions students have about this idea, each with its faulty reasoning in one line."
"For each misconception, write the exact answer a student holding it would give. No careless or random wrong answers."
"Label each option with the misconception it tests."
Then add your own: the mistakes you marked on last term's answer sheets are better evidence than anything the AI guesses. Our guide to using AI to find student misconceptions before you teach shows how to turn those notes into a list.
Worked example: one physics question
Stem: "A ball has left your hand and is still moving upward. Ignoring air resistance, which force or forces act on it?"
A. Gravity only, acting downward. (Correct.)
B. Gravity downward and a larger upward force from the throw. (Thinks motion needs a force in its direction.)
C. Gravity downward and an equal upward force. (Thinks a moving object needs balanced forces.)
D. No force until it starts to fall. (Thinks gravity acts only on falling objects.)
A first AI draft also offered "Air resistance only" and "Magnetic force". Both fail the one-sentence test: no student in the class would pick them, so they only make guessing easier.
How do you check each distractor before it goes in the paper?
Run every option through four checks:
Clearly wrong: could a strong student argue it is right? If yes, rewrite the stem or the option.
Named thinking: can you state the misconception behind it in one line?
Real students: have you seen this error in your class, or in a source you trust?
Different signal: does it point to a different idea from the other options? Two options from the same error waste a slot.
Check the key too, since AI sometimes marks a wrong option as correct: here is how to check answer keys of AI-made MCQs.
How many options should a question have?
Michael Rodriguez's meta-analysis of 80 years of research concluded that three-option items work without harming the quality of test scores, and let you ask more questions in the same time. If your format needs four, look for a fourth real misconception rather than padding. To spread good questions across a whole paper, see our post on planning a question paper blueprint with AI.
For students working through their wrong choice alone, Fermi's AI tutor notices where a student is stuck and decides whether to explain, ask or show. For a gap in the underlying idea, the Fermi Concept Library has a lesson on every concept in physics, chemistry, biology, mathematics and computing.
