On this page
When you speak to a phone and it types your words, it is solving a problem that looked almost hopeless in the 1960s. Sounds blur into each other, every speaker is different, and the same sound can belong to different words. Raj Reddy, born in Katoor, Andhra Pradesh, spent decades building computers that could follow continuous speech, and in 1994 shared the ACM A.M. Turing Award. This post covers his path, the main idea behind his systems, a short worked example on how speech systems are scored, and what students can take from his way of working.
Who is Raj Reddy?
The ACM Turing Award profile of Raj Reddy gives his full name as Dabbala Rajagopal Reddy, born on 13 June 1937 in Katoor, Andhra Pradesh. He attended the ZP High School at Sri Kalahasti in Chittoor district and took a degree in civil engineering from Guindy College of Engineering, Madras (now Anna University) in 1958. He then earned a master's degree at the University of New South Wales in Sydney in 1960 and worked for IBM in Australia.
In 1963 he joined Stanford University as a PhD student. In early 1964 he began a class project on speech recognition under John McCarthy, founder of the Stanford AI Lab, using the lab's PDP-1 computer to process speech waveforms. The ACM profile says he chose it because he was interested in natural languages. His 1966 PhD on speech recognition was the first PhD granted by Stanford's newly formed computer science department.
Why is understanding speech so hard for a computer?
In Speech recognition by machine: a review (1976), Reddy framed the problem around "sources of knowledge". A system has to use knowledge at several levels at once: acoustic (the sound wave), phonetic (which speech sounds are present), syntactic (which word orders are allowed) and semantic (what makes sense). Each level is unreliable on its own, so the system must generate guesses at one level and check them at the others.
A 1980 account of Hearsay-II by Erman, Hayes-Roth, Lesser and Reddy puts the difficulty well. Speech is produced through a long chain of transformations, from a speaker's intention to sound waves, so understanding it means effectively inverting that chain. At every step, ambiguity and uncertainty arise.
What was the blackboard idea?
Reddy moved to Carnegie Mellon University (CMU) in 1969. The ACM profile records that in the early 1970s the US agency DARPA funded a five-year research programme on speech understanding, and Reddy headed CMU's project. His group built systems called Hearsay-I, Hearsay-II, Harpy and Dragon.
Hearsay-II organised its work around a shared "blackboard". Separate programs, each working at a different level of abstraction, from sound up to words and phrases, posted guesses on the blackboard and built on each other's guesses. A focus-of-control mechanism, the 1980 paper explains, gave limited computing time first to "the most promising incremental actions". The ACM profile notes that this blackboard model for coordinating many knowledge sources was later adopted widely across applied artificial intelligence.
He shared the 1994 Turing Award with Edward Feigenbaum. The ACM citation reads: "For pioneering the design and construction of large scale artificial intelligence systems, demonstrating the practical importance and potential commercial impact of artificial intelligence technology." His Turing lecture was published as To dream the possible dream in Communications of the ACM in 1996.
Worked example: how do you score a speech recogniser?
A common measure today is word error rate: count the words the system substituted (S), deleted (D) and inserted (I), and divide by the number of words actually spoken (N). WER = (S + D + I) ÷ N. The sentences below are sample data for practice.
Spoken (N = 8 words): "the current in the circuit is two amperes"
System output: "the current in a circuit is to amperes please"
Line up the words. "the" became "a": 1 substitution. "two" became "to": another substitution. So S = 2.
Nothing was dropped, so D = 0.
"please" was added at the end, so I = 1.
WER = (2 + 0 + 1) ÷ 8 = 3 ÷ 8 = 0.375, or 37.5%.
Common mistakes:
Dividing by the output length. Always divide by the words actually spoken; here 8, not 9.
Ignoring insertions. Extra words are errors too, which is why WER can exceed 100%.
Treating homophones as correct. "To" for "two" sounds right but is a wrong word, exactly the kind of error Reddy's syntactic and semantic knowledge sources were meant to catch.
What else did he build?
The ACM profile lists him as founding director of CMU's Robotics Institute and Dean of its School of Computer Science from 1991 to 1999. He was awarded the Padma Bhushan in 2001. In India, the profile says, he chaired the governing council of IIIT Hyderabad, where he and colleagues developed Indian language processing, and helped found the Rajiv Gandhi University of Knowledge Technologies, which serves gifted rural youth. He told one interviewer, as the profile quotes: "All of my projects are interrelated. In order to solve any one of them, you have to solve all of them."
Our profile of Alan Turing covers the person the award is named after.
What can a student take from his way of working?
A class project can become a life's work. His started as one option among several that McCarthy suggested.
Break a hard problem into levels. Sounds, words, grammar and meaning are each easier to reason about than the whole.
Let weak clues combine. In physics too, a units check, a limiting case and a rough estimate together catch more errors than any one alone.
When you are stuck on a problem and do not know which level the error is at, Fermi's AI tutor notices where a student is stuck and decides whether to explain, ask or show.
Next step: write down a sentence, ask your phone to transcribe it, and work out the word error rate yourself.
