wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Problem

Framing

End-to-end ASR still depends on large transcribed corpora, while unlabeled speech is abundant. The paper closes this gap by masking latent audio frames and predicting quantized targets with a contrastive objective, reaching 1.8/3.3 WER on LibriSpeech test-clean/other and 4.8/8.2 WER with only 10 minutes of labels.

Currently Used Methods

Foundational

• @vaswaniAttentionAllNeed2017 — Transformer context modeling over long sequences.
  • Limitation in context: needs supervised targets; no speech-only pretraining objective.
• @devlinBERT2018 — masked prediction for contextual representation learning.
  • Limitation in context: discrete text tokens; raw speech lacks native tokenization.
• @bahdanauAttention2014 — neural sequence transduction for speech and translation.
  • Limitation in context: still relies on paired input-output supervision.
• @radfordWhisper2022 — large-scale speech recognition from weak supervision.
  • Limitation in context: uses labeled or pseudo-labeled transcripts, not pure self-supervision.
• @baevskiWav2Vec2_2020 — end-to-end masked contrastive speech pretraining with quantized targets.
  • Limitation in context: this paper is the method under review.

Proposed Method

Architecture

The model maps raw waveform $\mathbf{x}$ through a 7-layer temporal CNN to latent frames $\mathbf{z}_{1:T}$, masks sampled spans, then applies a Transformer to produce contextual states $\mathbf{c}_{1:T}$. A product quantizer converts unmasked latent frames into discrete targets $\mathbf{q}_t$ for contrastive prediction. BASE uses 12 Transformer blocks with model size 768 and 8 heads; LARGE uses 24 blocks, 1024, and 16 heads.

Loss / Objective

Pretraining combines masked contrastive prediction with a codebook-usage penalty.

Algorithm

Fine-tuning adds a linear projection over contextual states and optimizes CTC on labeled speech.

Training Procedure

• Mask starts: $p = 0.065$
• Mask span length: $M = 10$
• Masked fraction: about 49%
• Feature encoder: 7 conv blocks, 512 channels
• Conv kernels: $(10, 3, 3, 3, 3, 2, 2)$
• Conv strides: $(5, 2, 2, 2, 2, 2, 2)$
• Optimizer: Adam
• LR peak: $5 \times 10^{-4}$ BASE, $3 \times 10^{-4}$ LARGE
• Warmup: first 8% of updates
• Updates: 400k BASE, 250k LARGE, 600k LARGE on LV-60k
• Diversity weight: $\alpha = 0.1$
• Quantizer groups/codewords: $G = 2$, $V = 320$

Evaluation

Datasets

• LibriSpeech 960h labeled ASR
• Libri-light low-resource splits: 10 min, 1h, 10h
• LibriSpeech train-clean-100
• LS-960 unlabeled pretraining audio
• LV-60k unlabeled LibriVox audio, 53.2k hours
• TIMIT phoneme recognition

Metrics

• Word error rate (WER)
• Phoneme error rate (PER)

Headline results

• LibriSpeech 960h, LARGE + LV-60k + Transformer LM: test WER 1.8 / 3.3
• LibriSpeech 960h, LARGE + LV-60k + 4-gram LM: test WER 2.0 / 3.6
• 10 min labeled, pretrain on 53k hours: test WER 4.8 / 8.2
• 1 hour labeled: improves over prior 100h semi-supervised setups with 100\times fewer labels
• TIMIT: new SOTA PER with relative 23% / 29% gains on dev / test

Table 10: WER on LibriSpeech with all 960 hours labeled

Ablations

• Quantizing only contrastive targets beats feeding quantized inputs into the Transformer.
• Larger model size lowers WER across all low-resource and full-data setups.
• More unlabeled audio helps: LV-60k beats LS-960 at matched fine-tuning settings.
• Learned discrete latents align with phonetic categories, not arbitrary partitions.

Method Strengths and Weaknesses

Strengths

• Reaches 1.8/3.3 WER on LibriSpeech with full supervision after pretraining.
• Works in extreme low-label regimes: 4.8/8.2 WER with 10 minutes labeled.
• Learns quantization jointly, avoiding separate clustering pipelines.
• Improves TIMIT PER by large relative margins over prior systems.

Weaknesses

• Best results need 53.2k hours of unlabeled audio.
• Decoding quality still depends on external 4-gram or Transformer language models.
• LARGE model has 317M parameters, raising pretraining cost.
• Contrastive setup samples negatives from the same utterance only.

Suggestions from the authors

• Test whether discrete latents transfer beyond speech recognition.
• Study stronger links between learned units and phonetic structure.
• Extend pretraining to lower-resource languages lacking transcripts.
• Improve decoding without heavy external language models.

Links

Prior Papers

• @vaswaniAttentionAllNeed2017 — supplies the Transformer context network used to contextualize masked speech frames.
• @devlinBERT2018 — motivates span masking and contextual prediction as the pretraining template adapted to speech.

Further Papers

• @radfordWhisper2022 — large-scale speech modeling and recognition that benefits from robust pretrained speech representations.