On this page
Type “the” into a search box and almost every page on the web matches. Type “Coriolis” and you get a far smaller, far more focused set of pages. That difference, between a word that appears everywhere and a word that appears in only a few places, is the idea Karen Spärck Jones turned into arithmetic in 1972. Search engines still use it.
Who was Karen Spärck Jones?
Karen Spärck Jones was a British computer scientist who died on 4 April 2007, according to her obituary in Computational Linguistics by John Tait. Born in Huddersfield in 1935, she read History and then Philosophy at Cambridge, taught school briefly, and in 1957 joined the Cambridge Language Research Unit under Margaret Masterman. For a very long time she worked on short-term grants; Cambridge gave her a Readership only in 1994 and a personal chair in 1999.
Her real interest was language. After the ALPAC Report made machine translation hard to fund, she moved into information retrieval: finding the right document in a large pile. Tait records that in professional circles she was very often the only woman, and that she worked to interest young women and girls in computing, believing that “computing was too important to be left to men.”
What is inverse document frequency?
Her key paper, “A statistical interpretation of term specificity and its application in retrieval”, appeared in the Journal of Documentation, volume 28, in 1972. Its argument, as the 2004 reprint's abstract puts it, is that terms should be weighted by how often they occur in the collection, “so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms.” She tested this on three collections and found that the simple weighting improved retrieval.
Inverse document frequency (IDF) is a weight that grows as the number of documents containing a word shrinks; the common textbook form is log(N/n), where N is the number of documents and n is the number that contain the word (tf–idf, Wikipedia).
The insight is that a word's value depends on how rare it is in the collection, not on what it means.
How does IDF work? A worked example
Imagine a library of 1,000 pages of physics notes, so N = 1,000. Use log base 10 to keep the numbers clean.
“the” is on all 1,000 pages: IDF = log(1000/1000) = 0
“motion” is on 100 pages: IDF = log(1000/100) = 1
“projectile” is on 10 pages: IDF = log(1000/10) = 2
“Coriolis” is on 1 page: IDF = log(1000/1) = 3
Now search for “the Coriolis motion”. Page A contains “the” and “motion”: score 0 + 1 = 1. Page B contains “the” and “Coriolis”: score 0 + 3 = 3. Page B ranks first because it matched the word almost no other page has. Without the weights, both pages would tie at two matching words each.
Real systems add more: term frequency rewards a page that uses the word often (giving tf–idf), and modern formulas also adjust for page length.
How did Karen Spärck Jones shape search engines?
Tait's obituary says IDF “is the partial basis of all weighting schemes adopted by widely used Internet search engines” (Computational Linguistics, 2007). When The New York Times ran a belated obituary in 2019, its headline called her the woman “Who Established the Basis for Search Engines” (Wikipedia summary of the Overlooked obituary).
You can see the idea in working code today. Apache Lucene, an open-source search library, scores matches with BM25, and its documentation gives the IDF part as log(1 + (docCount − docFreq + 0.5)/(docFreq + 0.5)) (Lucene BM25Similarity). Elasticsearch describes its BM25 setting as a “TF/IDF based similarity” (Elasticsearch docs). The formula has grown, but her rule sits inside it.
IDF was never the whole story. Ranking pages by their links came later; see our post on Larry Page, Sergey Brin and the PageRank algorithm. Why a few words are so common and most are rare is described by Zipf's law of word frequency.
How can you use the IDF idea when you study?
Her rule works on your own searches too. Before you search for a doubt, try this checklist:
Cut the filler. “physics question help” is all common words; a search engine can do little with it.
Add the rare words from the question itself: “projectile”, “maximum range”, “45 degrees”.
Add one exact term from your textbook chapter, a word only that topic uses.
If results are still vague, swap a common word for a specific one (“speed” becomes “terminal velocity”).
If one of those rare terms is the very thing you don't understand, the Fermi Concept Library has a lesson for each concept in physics, chemistry, biology, mathematics and computing, with its place in the map. For deciding whether to search at all, see should you search Google or ask AI for a study doubt.
Next time you type a query, notice which word does the work. Karen Spärck Jones worked that out in 1972.
