WaveNet: A Generative Model for Raw Audio

Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu

2016 · arXiv

WaveNet: A Generative Model for Raw Audio

Problem

Framing

Raw-audio generation lacked a tractable model with long temporal context and TTS-grade perceptual quality. WaveNet closes this gap with an autoregressive convolutional stack that uses causal dilations and gated residual blocks. On Google TTS data, it reaches MOS $4.21 \pm 0.081$ for North American English.

Currently Used Methods

Direct antecedents

"Pixel Recurrent Neural Networks" — autoregressive discrete density modeling for images.
- Limitation in context: recurrent generation is slower and not tailored to long raw-audio context.
"Conditional Image Generation with PixelCNN Decoders" — gated autoregressive convolutions with discrete softmax outputs.
- Limitation in context: lacks dilated temporal stacks for audio-scale receptive fields.
"Generating Sequences With Recurrent Neural Networks" — sequence generation with recurrent hidden state.
- Limitation in context: training on very long raw waveforms is harder and less parallelizable.
"Deep Voice: Real-time Neural Text-to-Speech" — neural TTS pipeline with handoff between modules.
- Limitation in context: does not directly model waveform samples autoregressively.

Proposed Method

Architecture

WaveNet models raw waveform samples with a stack of causal $1$ -D convolutions. Dilation doubles across layers and repeats, e.g. $1,2,4,\ldots,512$ , giving exponential receptive-field growth. Each block uses a gated activation, residual connection, and skip path to a post-processing stack with ReLU, $1 \times 1$ convolutions, and softmax over $256$ $\mu$ -law levels.

Dilated causal convolution stack: input samples feed a causal 1-D convolutional hierarchy whose dilation grows from 1 to 8, expanding receptive field without pooling.

Loss / Objective

The model maximizes the autoregressive categorical likelihood over quantized samples.

p(\mathbf{x}) = \prod_{t=1}^{T} p(x_t \mid x_1, \ldots, x_{t-1})

z = \tanh\left(W_{f,k} \ast \mathbf{x}\right) \odot \sigma\left(W_{g,k} \ast \mathbf{x}\right)

Sampling Rule / Algorithm

Sampling proceeds left-to-right, one waveform value at a time, from the predicted categorical distribution.

x_t \sim p\left(x_t \mid x_1, \ldots, x_{t-1}\right)

Training Procedure

Output distribution: $256$ -way softmax over $8$ -bit $\mu$ -law encoded audio.
Convolution type: causal dilated convolutions.
Dilation schedule: repeated powers of two up to $512$ .
Conditioning: global speaker embedding; local linguistic features via learned upsampling.

Evaluation

Datasets

VCTK multi-speaker speech: $44$ hours, $109$ speakers.
Google TTS English: $24.6$ hours.
Google TTS Mandarin: $34.8$ hours.
MagnaTagATune music: about $200$ hours.
YouTube piano music: about $60$ hours.

Metrics

Mean opinion score (MOS) for TTS naturalness.
Paired subjective preference percentage.
Conditional log-likelihood for generative audio modeling.
Speech recognition error rate.

Headline results

Google TTS North American English: MOS $4.21 \pm 0.081$ .
Google TTS Mandarin Chinese: MOS $4.08 \pm 0.085$ .
English TTS, LSTM vs Concat baseline comparison: Concat preferred $63.6\%$ , LSTM $23.3\%$ .
English TTS, WaveNet vs best baseline: WaveNet preference exceeds baseline in paired tests.
Multi-speaker speech: one model generates distinct voices from speaker-ID conditioning.

Table 1: Subjective 5-scale MOS in naturalness

Speech samples	North American English	Mandarin Chinese
LSTM-RNN parametric	$3.67 \pm 0.098$	$3.79 \pm 0.084$
HMM-driven concatenative	$3.86 \pm 0.137$	$3.47 \pm 0.108$
WaveNet (L+F)	4.21 \pm 0.081	4.08 \pm 0.085

Ablations

Conditioning type: local linguistic features plus $f_0$ beats weaker conditioning.
Speaker conditioning: global one-hot speaker input enables one model for many voices.
Receptive field size: larger dilated context is the core mechanism for long-range audio structure.
Output distribution: categorical softmax beats mixture-density style alternatives for this setting.

Method Strengths and Weaknesses

Strengths

Direct waveform likelihood avoids handcrafted vocoder assumptions.
Dilated causal convolutions give large context without recurrence.
TTS MOS beats strong LSTM and concatenative baselines.
Global and local conditioning support multi-speaker and text-conditioned generation.

Weaknesses

Autoregressive synthesis is inherently sequential at sample rate.
Evaluation leans heavily on subjective listening tests.
Training hyperparameters are sparsely reported.
Music and speech-recognition results are less emphasized than TTS.

Suggestions from the authors

Apply the model to broader audio tasks such as enhancement and source separation.
Improve conditioning mechanisms for richer text-to-speech control.
Extend waveform modeling to more diverse music generation settings.
Adapt the architecture further for discriminative speech recognition.

WaveNet: A Generative Model for Raw Audio

WaveNet: A Generative Model for Raw Audio

Problem

Framing

Currently Used Methods

Direct antecedents

Proposed Method

Architecture

Loss / Objective

Sampling Rule / Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers