On this page
You tap the microphone, say "acceleration is nine point eight metres per second squared", and words appear before you finish. Here is what happens in that second, and why it sometimes goes wrong.
How does speech to text work in phones?
It starts with physics. Your phone's microphone turns pressure waves into a changing electric signal (we explain that step in how a microphone turns sound waves into electric current).
Whisper, an open speech recogniser from OpenAI, shows the general idea; its research paper describes these steps:
Sample. The audio is measured 16,000 times a second.
Slice. The signal is cut into overlapping 25-millisecond windows, a new one every 10 milliseconds.
Map. Each window becomes a column of 80 numbers saying how loud each band of frequencies is. Placed side by side, the columns form a log-Mel spectrogram, a picture of sound like the simplified one in the diagram above.
Guess. A neural network (an encoder-decoder Transformer) reads the spectrogram and writes out the most likely text, one piece at a time.
Whisper learned this from 680,000 hours of audio paired with transcripts, according to the Whisper paper.
Your phone may use a different model, but the steps are broadly similar. In 2019 Google began running Gboard's speech recogniser on the phone itself, starting with Pixel phones in American English: an 80 MB model that works offline and outputs words character by character, as if someone were typing what you say.
Why does voice typing get some accents wrong?
Accuracy is usually reported as word error rate: roughly, the fraction of words the system gets wrong.
In a 2020 study in PNAS, Allison Koenecke and colleagues tested five commercial systems, from Amazon, Apple, Google, IBM and Microsoft, on interviews with 42 white and 73 Black speakers in five US cities. The average word error rate was 0.35 for Black speakers and 0.19 for white speakers. The gap was just as large on identical phrases, and the authors traced it to the systems' acoustic models, proposing more diverse training data among other fixes.
The Whisper paper shows why training data matters so much: across languages, a language's word error rate roughly halves for every 16-fold increase in its training data (Whisper paper). The PNAS study covered only US speakers, so it gives no number for Indian English. But if your accent, or your mix of Hindi and English, is rarer in the training audio, you may see more errors. That is a limit of the data, not a flaw in how you speak.
Why do noise and science words cause errors?
Noise. Background sound blurs the spectrogram. When the Whisper team added pub-style chatter to test recordings, all the models they compared got worse as the noise grew louder (Whisper paper). A ceiling fan or traffic can do the same to your phone.
Rare words. Whisper's decoder is sometimes given the earlier transcript "in the hope that it will learn to use longer-range text context to resolve ambiguous audio", in the paper's words. Context helps everyday sentences, but a rare term can be swapped for a common word that sounds similar. Watch for slips like "sigma bond" becoming "stigma bond".
Symbols. Phones write words, not equations, so "v equals u plus a t" may come back with "you" instead of "u".
How can you dictate science notes more reliably?
Use this five-step routine:
Cut the noise. Switch off the fan, close the window, and keep the phone the same distance from your mouth.
Speak in short, complete sentences. A whole sentence gives the model context.
Say the punctuation. On Android's Gboard, you can say "period", "comma", "question mark", "exclamation point", "new line" and "new paragraph", according to Google's voice typing help page, which notes that punctuation might not be available in all languages.
Say maths as words, add symbols later. Say "v equals u plus a t" and fix the symbols when you review.
Keep a fix list. Note the terms your phone keeps mishearing, often chemical names, scientists' names and Greek letters, and type those instead.
Here is a worked example. Say you are dictating a note on Newton's second law. Speak it like this:
"Newton's second law, new line. Net force equals mass times acceleration, period. Force in newtons, mass in kilograms, period."
In your review pass:
turn "net force equals mass times acceleration" into F = ma
check that "newtons" did not become "Newton's"
If a line makes no sense when you read it back, the gap may be in the idea rather than the transcript. The Fermi Concept Library has a lesson for every concept in physics, chemistry, biology, mathematics and computing.
If you would rather talk a topic through, see our guide to using a voice mode AI tutor for revision; Fermi's AI tutor can work out what you already know and build a plan for the session.
