How does speech-to-text like Whisper work?

Whisper is an encoder-decoder transformer trained for automatic speech recognition. The audio is converted into a spectrogram (a time-frequency image of the sound), the encoder turns that into representations, and the decoder autoregressively generates the text transcript token by token, much like a translation model. It was trained on a very large, diverse set of weakly-labelled audio from the web across many languages, which makes it robust to accents, background noise, and domains without task-specific tuning, and lets it also translate and identify language. Limitations: it can hallucinate text during silence or noise, struggles with heavy overlap between speakers, and doesn't do speaker diarisation on its own. It's a strong general-purpose default that you can fine-tune for a specific domain's vocabulary.