Robust Speech Recognition via Large-Scale Weak Supervision

Problem

Framing

Supervised ASR had strong in-domain accuracy but weak transfer, brittle multilingual coverage, and fragmented task pipelines. Whisper closes this with a single encoder-decoder Transformer trained on 680k hours of weakly supervised web audio, using task-specifying text tokens for transcription, translation, language ID, and VAD.

Currently Used Methods

Foundational

• @oordWaveNet2016 — autoregressive raw-audio generation shows sequence modeling at waveform scale.
  • Limitation in context: not an end-to-end multilingual ASR transfer recipe.
• @baevskiWav2Vec2_2020 — self-supervised speech pretraining with strong fine-tuned ASR.
  • Limitation in context: still relies on supervised adaptation and task-specific pipelines.
• "Attention Is All You Need" — Transformer sequence modeling enables large-scale encoder-decoder learning.
  • Limitation in context: does not specify speech multitask prompting or weak-supervision scaling.
• "Scaling Laws for Transfer" — scaling data and parameters improves weakly supervised transfer.
  • Limitation in context: speech had little evidence at 680k-hour multilingual scale.

Proposed Method

Architecture

Whisper uses a sequence-to-sequence Transformer over 30-second 80-channel log-Mel spectrograms. The encoder applies two Conv1D+GELU front-end layers, sinusoidal positions, and Transformer blocks; the decoder uses learned positions, causal self-attention, and cross-attention into the encoder. Model sizes range from Tiny to Large, with 4 to 32 layers and 39M to 1550M parameters.

Loss / Objective

Training is standard next-token prediction over the multitask token stream.

Algorithm

Task control is done by prefix tokens that specify language, task, timestamp behavior, and optional prior context.

Training Procedure

• Training data: 680k hours.
• Audio context: 30 seconds.
• Batch size: 256.
• Updates: 1,048,576.
• Warmup updates: 2,048.
• Max grad norm: 1.0.
• Learning rates: Tiny 1.5 $\times$ 10^{-3}, Base 1 $\times$ 10^{-3}, Small 5 $\times$ 10^{-4}, Medium 2.5 $\times$ 10^{-4}, Large 1.75 $\times$ 10^{-4}.

Evaluation

Datasets

• LibriSpeech test-clean, test-other.
• TED-LIUM 3.
• Common Voice 5.1 and 9.
• CHiME-6.
• CallHome, Switchboard.
• Fleurs.
• VoxPopuli.
• CoVoST 2.

Metrics

• Word error rate (WER).
• BLEU for $X \rightarrow \text{en}$ translation.
• Language identification accuracy.
• Robustness under additive noise and long-form transcription.

Headline results

• LibriSpeech test-clean: WER 2.7.
• LibriSpeech test-other: WER 5.9.
• CoVoST2 $X \rightarrow \text{en}$: beats prior zero-shot systems in overall, medium, and low-resource groups.
• Fleurs multilingual ASR: speech-recognition supervision strongly predicts zero-shot WER with $r^2 = 0.83$.
• Fleurs translation: translated-audio hours predict zero-shot BLEU weakly with $r^2 = 0.24$.

Ablations

• Model size: scaling improves English, multilingual, and translation performance smoothly.
• Dataset size: gains persist up to the full 680k-hour corpus, then diminish.
• Noise level: Whisper degrades more gracefully than competing systems under white and pub noise.
• Task mixture: multitask prompting supports one model across ASR, translation, language ID, and VAD.

Method Strengths and Weaknesses

Strengths

• One model covers ASR, translation, language ID, and VAD through prompting.
• 680k-hour weak supervision yields strong zero-shot transfer across many datasets.
• LibriSpeech reaches 2.7 WER on test-clean without a language model.
• Noise robustness exceeds competing systems across SNR conditions.

Weaknesses

• Translation transfer is weaker than ASR transfer; Fleurs BLEU correlation is only $r^2 = 0.24$.
• Language identification is not competitive with supervised specialists.
• Decoder can emit speaker names or hallucinatory text on long-form audio.
• Many remaining errors are linguistic and world-knowledge failures, not acoustics.

Suggestions from the authors

• Improve decoding to reduce linguistic and world-knowledge errors.
• Study language-model fusion for pretrained speech encoders and Whisper-like decoders.
• Add auxiliary objectives beyond next-token prediction.
• Improve long-form transcription behavior and suppress speaker-name hallucinations.

Links

Prior Papers

• @baevskiWav2Vec2_2020 — strong speech representation pretraining baseline that Whisper challenges with weakly supervised multitask transfer.
• @oordWaveNet2016 — early large-scale sequence modeling for speech audio, useful as historical context for end-to-end speech systems.

Further Papers

• @alayracFlamingo2022 — shares the theme of large-scale weakly supervised multimodal transfer with prompted downstream behavior.
• @luUnifiedIO2_2024 — extends the single-model, many-task paradigm to broader multimodal sequence transduction.