On this page
The support vector machine (SVM) is a classic machine-learning tool: give it labelled examples from two groups and it draws the boundary with the widest possible gap between them. Its central figure is Vladimir Vapnik, a Soviet-trained mathematician.
Who invented the support vector machine?
The SVM came together at AT&T Bell Laboratories in the early 1990s. In 1992, Bernhard Boser, Isabelle Guyon and Vapnik presented a training algorithm for optimal margin classifiers at the Workshop on Computational Learning Theory. In 1995, Corinna Cortes and Vapnik published "Support-vector networks" in the journal Machine Learning, extending the method to training data that cannot be separated without errors.
The 1995 paper says the optimal hyperplane for separable classes was worked out in 1965, in Vapnik's earlier work. So the fair answer is a Bell Labs team building on Vapnik's research from the 1960s.
Who was Vladimir Vapnik?
Vapnik was born on 6 December 1936, according to his profile at the Institute of Control Sciences in Moscow. The Franklin Institute records a 1958 master's in mathematics from Uzbek State University and a 1964 doctorate in statistics from the Institute of Control Sciences.
In Moscow he worked with Alexey Chervonenkis. The Institute of Control Sciences says the pair developed the "generalized portrait method" for pattern recognition between 1962 and 1971, and in 1968 proved conditions for relative frequencies to converge uniformly to probabilities. Their paper on that result appeared in 1971 in Theory of Probability and Its Applications. In plain words, they asked: when does doing well on the examples you have seen guarantee doing well on new ones? The Franklin Institute credits the resulting Vapnik-Chervonenkis (VC) theory with establishing the foundations of contemporary machine learning theory.
In 1990, the Institute of Control Sciences says, Vapnik moved to the United States and joined AT&T Bell Laboratories, where he grew the generalized portrait method into support vector machines (the Franklin Institute says 1991). In 2012 the Franklin Institute awarded him the Benjamin Franklin Medal in Computer and Cognitive Science.
What does a support vector machine actually do?
Picture red squares and blue dots on graph paper. Many straight lines separate them; Vapnik's answer to which one to trust is the line with the widest margin. Cortes and Vapnik define the optimal hyperplane as the linear decision function with maximal margin between the vectors of the two classes (Cortes and Vapnik, 1995). Only a small part of the training data, the points nearest the boundary, decides where it goes; these are the support vectors.
Why care? For data separated without errors, the paper shows the expected chance of a mistake on a new example is at most the expected number of support vectors divided by the number of training examples. It reports that this ratio can be as low as 0.03 for real problems.
The 1992 paper added a shortcut: compare two points first (say, by their dot product) and then transform that single number, instead of moving every point into a huge feature space. This lets SVMs draw curved, polynomial boundaries.
How do you find the widest margin by hand?
You need only Class 11 coordinate geometry: the distance of a point from a line, |ax + by + c| / √(a² + b²).
Take three points of class A: (0, 0), (1, 0) and (0, 1). Take three points of class B: (2, 2), (3, 2) and (2, 3).
Find the facing points: (1, 0) and (0, 1) lie on x + y = 1, and (2, 2) on x + y = 4.
Put the boundary halfway: x + y = 2.5. Each gap is 1.5/√2, about 1.06, so the street is about 2.12 wide.
Compare a worse line, x + y = 2. Its gap to (1, 0) is only 1/√2, about 0.71, so a new class A point at (1.2, 0.9) lands on its wrong side, but on the right side of x + y = 2.5.
Write it the SVM way, scaled so support vectors give exactly +1 or -1: f(x, y) = (2/3)x + (2/3)y - 5/3. Check: f(2, 2) = +1 and f(1, 0) = -1. A point is class B if f > 0, so (2, 1), with f = 1/3, is class B.
The points (0, 0), (3, 2) and (2, 3) never mattered. That is the support-vector idea.
What can a student take from Vapnik's work?
The maths of an SVM is in your syllabus: vectors, dot products, the distance of a point from a line, and maximising under a condition.
Study the border cases. The mock-test questions you nearly got wrong are your support vectors: they show where your understanding is thinnest.
If the distance formula or dot products feel shaky, the Fermi Concept Library has a lesson for every concept in mathematics. To check your working on a problem like the one above, the Fermi Problem Library has a step-by-step solver that explains the reasoning rather than printing an answer.
For more of this story, read about Frank Rosenblatt and the perceptron, Yann LeCun and convolutional networks and Andrey Kolmogorov's axioms of probability.
