On this page
You learnt to ride a cycle without a manual: you wobbled, fell and kept what worked. Reinforcement learning gives a computer the same deal: no answer key, only points and penalties.
What is reinforcement learning, in simple words?
In the standard textbook by Richard Sutton and Andrew Barto, Reinforcement Learning: An Introduction, reinforcement learning means learning "how to map situations to actions" so as to maximise a numerical reward signal. The same book says the learner "is not told which actions to take" but must discover which actions yield the most reward "by trying them out."
That differs from supervised learning, where a model studies examples labelled with the right answer; our explainer on AI vs machine learning vs deep learning gives one example of each. The learner is the agent, where it stands is the state, what it does is the action, and the score that comes back is the reward.
How does an AI learn a grid game by reward and penalty?
Picture a 4×4 board. The agent starts bottom-left. The top-right square holds a prize worth +10, the square diagonally below-left of it is a pit worth −10, and there is one wall. Every move that does not end the game costs −1, so wandering is punished.
At first the agent moves at random, falling in the pit and taking detours. After each move it updates its estimate of that move's value from that square. A widely used rule, Q-learning, comes from Christopher Watkins's 1989 work and was set out with Peter Dayan in Machine Learning in 1992. In plain words:
new estimate = old estimate + step size × (reward now + best estimate from the next square − old estimate)
A worked update: every estimate starts at 0, the step size is 0.5, and the agent stands directly below the pit. It moves up and falls in: reward −10, and there is no next square. New estimate = 0 + 0.5 × (−10 − 0) = −5. Now "up" looks bad there, and positive estimates spread backwards from the prize the same way.
Once learning settles, each square's number is what the best route from there is worth, as the diagram above shows. The start is worth 5: six moves, five costing −1 and the last earning +10.
Sutton and Barto note that an agent "has to exploit what it already knows in order to obtain reward, but it also has to explore." So it mostly takes the best-known move, sometimes a random one.
The recipe scales up. A deep Q-network, given "only the pixels and the game score" as inputs, reached a level comparable to a professional human games tester across a set of 49 Atari 2600 games (Nature, 2015). Earlier still, Arthur Samuel's checkers program learned from play.
Can you try the trial-and-error method yourself?
Yes, with paper, a pencil and a dice.
Draw a 4×4 grid, counting rows from the top: Start bottom-left, Prize (+10) top-right, Pit (−10) in row two, column three, Wall in row three, column two.
Roll to move: 1 up, 2 down, 3 left, 4 right, 5 or 6 roll again. If you hit the wall or edge, stay put and still pay −1.
Score −1 per move, except the last: +10 onto the prize, −10 into the pit. The game ends at either.
After each game, write in every square you visited the points you collected from there to the end. Keep the bigger number if one is already there.
After five games, roll once per move: on 1 to 5, step to the neighbour with the highest number; on a 6, roll again for a random direction.
Stop when your route from Start scores 5, the best possible.
For ideas that are easier to see than to read, the Fermi Simulation Lab has interactive simulations.
Where do chatbots use reinforcement learning?
Chatbots can be tuned with reinforcement learning from human feedback, or RLHF. In the InstructGPT paper, Long Ouyang and colleagues collected rankings of model answers and used them to further fine-tune GPT-3 with reinforcement learning. Here the reward comes from what people preferred, not +10 for a square. In their human evaluations, answers from the 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3, despite it having 100 times fewer parameters.
Reward has a catch: an agent learns what is rewarded, not what you meant. A 2023 study of sycophancy by Mrinank Sharma and colleagues found sycophancy, a pull towards agreeing with the user, in five AI assistants. In the human preference data they analysed, "when a response matches a user's views, it is more likely to be preferred." So when you study with a chatbot, ask it to find the first wrong step in your working rather than "Is this right?", and see how to test whether an AI is just agreeing with you.
We built Fermi's AI tutor to teach rather than only answer: it notices where you are stuck, decides whether to explain, ask or show, and checks understanding as it goes.
Play the paper game once tonight, and next time you hear an AI "learned" something, ask what it was rewarded for.
