How Speech-to-Text Works: From Sound Waves to Transcripts

Speech recognition went from a brittle research problem to a solved-ish commodity in about three years. Here is the pipeline that made it work — and where it still breaks.

By AIWI EditorialEdited by Chayanika Divekar4 min readPublished Jul 15, 2026

Corroborated2 sources

Key Takeaways

Modern speech-to-text converts audio into a spectrogram — a picture of sound — then treats transcription as a sequence-to-sequence problem, the same shape as translation.
OpenAI's Whisper trained on 680,000 hours of weakly-supervised, messy real-world audio, which is exactly why it handles accents and background noise better than cleaner-trained predecessors.
Scale of diverse data beat architectural cleverness: the model is a fairly ordinary encoder-decoder transformer.
The classic failure mode is hallucination on silence — given nothing to transcribe, the decoder invents plausible text.
Out of the box these models do not tell you who spoke; speaker diarization is a separate system bolted alongside.

Ten years ago, speech recognition was a research problem with a demo caveat: it worked, in a quiet room, in a familiar accent, on a good microphone. Today it’s a commodity API that handles a noisy café and a heavy accent without blinking. The interesting part is how that happened — because the answer isn’t a clever new architecture.

Step 1: sound becomes a picture

Raw audio is a waveform — amplitude sampled thousands of times a second. It’s a terrible thing to feed a neural network directly: enormous, and the information you care about (which sounds are being made) is tangled up in the frequencies rather than the raw amplitude.

So the first step converts the waveform into a spectrogram: a 2D map with time on one axis, frequency on the other, and brightness showing how much energy is present at that frequency at that moment. Usually it’s a log-Mel spectrogram, where the frequency axis is warped to match human hearing — finer resolution at low frequencies, coarser at high, because that’s where speech information actually lives.

The important shift here is conceptual: audio has become an image. And we know how to feed images to neural networks.

Step 2: transcription is translation

Once audio is a spectrogram, transcription becomes a sequence-to-sequence problem — turn one sequence (audio frames) into another (text tokens). That’s the exact shape of machine translation, so it uses the exact same tool: an encoder-decoder transformer.

The encoder reads the whole spectrogram and builds a representation of what was said, with attention letting every moment of audio be interpreted in the context of every other moment.
The decoder generates text tokens one at a time, attending to the encoder’s output — a language model conditioned on the audio.

That decoder-is-a-language-model detail is worth holding onto. It’s the source of both the biggest strength (the model produces fluent, correctly-punctuated text because it knows what English sounds like as language) and the worst failure mode (see below).

Step 3: the actual secret — messy data at scale

Here’s the part that matters. Whisper, the model that made robust speech-to-text ordinary, has a fairly unremarkable architecture. What was remarkable was the training set: 680,000 hours of audio scraped from the web with weak supervision — imperfect, machine-generated, real-world transcript pairs, in dozens of languages, rather than a small hand-curated corpus.

Previous systems trained on clean, carefully-labeled speech and were superb at clean, carefully-labeled speech. Whisper trained on the mess: accents, crosstalk, bad microphones, background music, code-switching. So when it meets a noisy café, that isn’t out-of-distribution — that is the distribution.

This is the same lesson LLM pretraining taught, arriving in a different field: diverse data at scale beat architectural cleverness. Robustness wasn’t engineered; it was trained in.

Because the training set was multilingual, one model also transcribes many languages and translates speech directly to English — capabilities that fell out of the data rather than being designed.

The honest limits

Hallucination on silence. The signature failure. Feed a Whisper-style model a segment with no speech and it may confidently produce a sentence — often something from its training set’s ambient noise, like a subtitle sign-off. The decoder is a language model doing what language models do: generating plausible text. Silence gives it nothing to anchor on, so it generates anyway. Real deployments run voice activity detection first and never send silent segments to the model.

No speaker labels. The output is words, not “who said what.” Speaker diarization is a separate system; if you need meeting transcripts with names, you’re integrating two models.

Domain vocabulary. Proper nouns, drug names, internal jargon, and ticker symbols are guessed by phonetic plausibility. The model will produce something that sounds right and is wrong. Prompting with a vocabulary list or post-correcting against a term glossary is standard practice.

Confidence is not calibrated. Fluent output is not accurate output. A transcript that reads perfectly can be wrong in exactly the words you care about — which is why anything consequential (clinical, legal) keeps a human on the loop.

The bottom line

Speech-to-text is a good case study in how a hard problem quietly got solved: turn sound into a picture, treat transcription as translation, then train on a genuinely representative mess of real audio rather than a clean lab dataset. The architecture is borrowed; the robustness came from the data.

If you’re building on it, the two things to design for are the ones the demos skip — cut silence before it reaches the model, and don’t trust fluency as a proxy for accuracy. For the multimodal picture this fits into, see how CLIP-style multimodal search works; for the transformer machinery underneath, start with how transformers work.

Was this helpful?

Sources & Further Reading

Robust Speech Recognition via Large-Scale Weak Supervision — arXiv (OpenAI)
Introducing Whisper — OpenAI

Frequently Asked Questions

Why do speech-to-text models hallucinate on silence?

Because the decoder is a language model. It is trained to produce the most likely next token given the audio and the text so far — and when the audio contains nothing to anchor on, 'most likely text' can still be a fluent, confident sentence that was never spoken. Silence removes the evidence but not the model's urge to generate. Voice activity detection that skips silent segments before they reach the model is the standard mitigation.

Does speech-to-text identify who is speaking?

Not on its own. Transcription and speaker diarization ('who spoke when') are different problems. A Whisper-style model gives you the words; a separate diarization system clusters voice embeddings into speakers, and the two outputs are merged afterwards. If you need speaker labels, plan for two components, not one.

Multimodal AI Goes Mainstream: One Model, Many Senses — Jul 3, 2026
Runway launches AI model router as generative media gets crowded — Jul 23, 2026
Here’s what Samsung’s smart glasses actually look like — Jul 22, 2026
Meta made its own AI detection system. It should have just used Google’s — Jul 22, 2026

#speech #audio #transformers #multimodal #education