On this page
Chatbots can feel like magic, and magic is hard to question. In this 40-minute activity, a class builds a working next-word predictor by hand with paper, pencils and a paragraph, then gives a chatbot the same opening words and compares. Students come away knowing what "predicting the next word" means, and why a bigger model writes better text without the idea changing.
How does AI predict the next word?
A language model predicts upcoming words by giving a probability to each possible next word, according to chapter 3 of Jurafsky and Martin's Speech and Language Processing. The simplest kind is a bigram model, which guesses the next word using only the one word before it. You estimate its probabilities by counting: the probability that word B follows word A is the number of times "A B" appears divided by the number of times A appears, the same textbook explains. It also notes that large language models are built by training them to predict words, and that in practice language models usually work with tokens (pieces of words) rather than whole words.
The idea is old. In his 1948 paper A Mathematical Theory of Communication, Claude Shannon built approximations to English with a book rather than a computer: open it at random, read until you find the current word, and write down the word that comes after it. His second-order word approximation produced lines like "the head and in frontal attack on an English writer", which reads almost like sense for a few words at a time. Your class will repeat a version of it.
How do you run the activity in class?
Use this paragraph, or ask students to write five short sentences of their own:
"The rain fell on the roof. The rain fell on the road. The children ran to the window. The children saw the rain on the glass. The rain stopped and the children ran out."
Ignore capitals and full stops, and only pair words inside the same sentence.
For each word, list every word that comes straight after it, with tally marks (10 minutes).
Turn tallies into fractions: count of the pair divided by the total for that first word (5 minutes).
For each word, write its followers on paper slips, one slip per tally, and put them in an envelope labelled with that word.
Start with "the", draw a slip, write down the word, then draw from that word's envelope. Put each slip back after drawing. Stop at 12 words or an empty envelope (10 minutes).
Now try the "always pick the most likely word" rule instead of drawing.
Compare with a chatbot (10 minutes, next section).
What do the counts look like for this paragraph?
In the paragraph above, "the" is followed by another word 11 times:
rain: 4 of 11, about 36%
children: 3 of 11, about 27%
roof, road, window, glass: 1 of 11 each, about 9%
After "rain" comes "fell" 2 of 4 times, with "on" and "stopped" once each. "Fell" is always followed by "on", and "on" is always followed by "the".
Drawing slips can give "the rain stopped and the children saw the rain on the road". That sentence is not in the paragraph, yet every pair in it is. The "most likely word" rule gives "the rain fell on the rain fell on the...", stuck in a loop.
What happens when you compare it with a chatbot?
Type "The rain fell on the" into any chatbot and ask it to continue for one sentence. Then ask the class:
What is the same? Both produce one word (or word piece) at a time, chosen from what the text before suggests.
Why does the chatbot not loop? Our model looks back one word and knows 15 words. A large language model is trained on a vast amount of text and uses much more of what came before.
Can the chatbot be confidently wrong? Yes. It predicts likely text, not checked facts, which is why sources and maths need checking. Our guides on why AI chatbots get maths wrong and checking references AI makes up give students a routine for that.
What should students write down at the end?
Ask for three sentences: what a language model predicts, how they got probabilities from counts, and one difference from the chatbot. For the next layer, our explainer on how large language models work picks up from here.
The maths underneath is conditional probability. If the fractions in step 3 felt shaky, the Fermi Concept Library has its own lesson for every concept in mathematics. And if you want to see how an AI built to teach differs from one that only continues text, Fermi's AI tutor works out what a student already knows and checks understanding as it goes.
