On this page
In 1989, Yann LeCun and colleagues at AT&T Bell Laboratories showed that one neural network could learn to read handwritten ZIP code digits (US postal codes), going from the normalised image of each digit to its classification (LeCun et al., Neural Computation, 1989). By 1998 a later network, LeNet-5, was the core of a commercial cheque reader.
Who is Yann LeCun, and what did he build?
The ACM's 2019 announcement says that "in the late 1980s, while working at the University of Toronto and Bell Labs, LeCun was the first to train a convolutional neural network system on images of handwritten digits" (ACM release via EurekAlert). The same release names LeCun, Yoshua Bengio and Geoffrey Hinton as winners of the 2018 ACM A.M. Turing Award, which carries a $1 million prize, and calls the convolutional neural networks LeCun developed "a foundational principle in the field".
The 1989 paper argued that a network generalises better when constraints from the task are built into its wiring (LeCun et al., 1989). One network learned the whole job, from image to digit, using backpropagation, the method explained in our story of how backpropagation taught neural networks to learn.
What is a convolutional neural network?
A convolutional neural network is a neural network built on three ideas: local receptive fields, shared weights and sub-sampling, as set out by LeCun, Bottou, Bengio and Haffner (Proceedings of the IEEE, 1998).
Local receptive fields: each unit looks at a small patch, such as a 5×5 square of pixels, so it can pick up edges, end-points and corners.
Shared weights: every unit in a feature map uses the same 25 weights and bias, so one feature detector slides across the whole image. That sliding operation is a convolution, which gives the network its name.
Sub-sampling: averaging 2×2 blocks halves the resolution and makes the output less sensitive to small shifts and distortions.
The paper traces the local-patch idea to the Perceptron of the early 1960s, around the time Hubel and Wiesel found orientation-selective neurons in the cat's visual system.
How did LeNet-5 read a handwritten digit?
The 1998 paper describes LeNet-5 as seven layers with trainable weights, not counting the 32×32 pixel input image (LeCun et al., 1998). Follow one digit through it:
Input: a 32×32 image.
C1: 6 feature maps of 28×28, each made by sliding a 5×5 filter.
S2: 6 maps of 14×14, after 2×2 averaging.
C3: 16 maps of 10×10, again from 5×5 filters.
S4: 16 maps of 5×5.
C5 and F6: 120 units, then 84 units.
Output: 10 units, one for each digit from 0 to 9.
LeNet-5 has 340,908 connections but only 60,000 trainable parameters, because of weight sharing (LeCun et al., 1998).
Worked example: check the paper's numbers yourself
Verify the first layer with school arithmetic (LeCun et al., 1998).
Size: a 5×5 filter fits in 32 − 5 + 1 = 28 positions along each side, so each map is 28×28.
Parameters: each filter has 25 weights plus 1 bias, so 6 filters need 6 × 26 = 156 numbers. The paper lists exactly 156.
Connections: the 6 × 28 × 28 = 4,704 units each use 26 inputs, so 4,704 × 26 = 122,304 connections, again the paper's figure.
Without sharing: if every unit kept its own 26 numbers, C1 would need 122,304 parameters instead of 156. Wiring each unit to all 1,024 input pixels would need 1,024 × 4,704 = 4,816,896 weights, plus 4,704 biases.
Now try C3: 14 − 5 + 1 = ?
How accurate was LeNet-5, and where was it used?
LeCun's team built MNIST, a database of handwritten digits mixed from two US National Institute of Standards and Technology sets, with 60,000 training images and 10,000 test images (LeCun et al., 1998). LeNet-5's test error was 0.95%, falling to 0.8% when 540,000 artificially distorted digits were added to the training set.
The abstract of the same paper says a cheque-reading system built on convolutional networks "is deployed commercially and reads several million checks per day" (LeCun et al., 1998). The introduction says LeNet-5 is its core, in NCR Corporation's cheque recognition systems used by several banks in the United States.
How can you try LeCun's idea by hand?
A 20-minute exercise with pencil and paper:
Draw a 6×6 grid of 0s with one vertical line of 1s near the middle.
Slide a 3×3 filter whose columns are −1, 0 and +1 across it, multiplying and adding at each stop. You get a 4×4 map with large positive and negative values on either side of the line.
Average each 2×2 block, then shift the line by one pixel and compare the two results.
Count the weights you used, then count how many you would need without sharing.
If matrices or convolution feel shaky, the Fermi Concept Library covers every concept in mathematics and computing, each with its own lesson and its place in the map, so you can fill the gap first.
Next, read how Fei-Fei Li's ImageNet changed artificial intelligence, or start with AI vs machine learning vs deep learning: one example each.
