Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko

2020 · NeurIPS

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Problem

Framing

Contrastive SSL depended on negative pairs, large batches, or memory banks to avoid collapse. BYOL replaces negatives with bootstrap regression between two augmented views, using an online predictor and an EMA target network. On ImageNet linear evaluation, it reaches 74.3% top-1 with ResNet-50 and 79.6% with ResNet-200 (2 $\times$ ).

Currently Used Methods

Foundational

@chenSimCLR2020 — contrastive learning with strong augmentations and large-batch negatives.
- Limitation in context: needs negatives and degrades sharply with smaller batches.
Momentum Contrast for Unsupervised Visual Representation Learning — momentum encoder and memory bank for contrastive SSL.
- Limitation in context: still depends on negative-pair discrimination.
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments — cross-view prediction via online clustering.
- Limitation in context: bootstraps cluster assignments, not continuous representations.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results — EMA teacher stabilizes student targets.
- Limitation in context: needs labels; removing supervision collapses in this setting.

Proposed Method

Loss / Objective

BYOL uses two branches. The online branch is encoder $f_{\theta}$ , projector $g_{\theta}$ , predictor $q_{\theta}$ ; the target branch is encoder $f_{\xi}$ and projector $g_{\xi}$ with no predictor. The target parameters follow an EMA of the online weights, and only $f_{\theta}$ is kept after training.

Architecture diagram: an input image is augmented into two views; the online path applies encoder, projector, and predictor, while the target path applies encoder and projector with stop-gradient before the similarity loss.

Loss / Objective

The objective regresses the normalized online prediction to the normalized target projection, then symmetrizes across the two views.

\mathcal{L}_{\theta,\xi} = \left\| \frac{q_{\theta}(z_{\theta})}{\|q_{\theta}(z_{\theta})\|_2} - \frac{z_{\xi}'}{\|z_{\xi}'\|_2} \right\|_2^2 = 2 - 2 \cdot \frac{\langle q_{\theta}(z_{\theta}), z_{\xi}' \rangle}{\|q_{\theta}(z_{\theta})\|_2 \cdot \|z_{\xi}'\|_2}

\mathcal{L}^{\mathrm{BYOL}}_{\theta,\xi} = \mathcal{L}_{\theta,\xi} + \tilde{\mathcal{L}}_{\theta,\xi}

Sampling Rule / Algorithm

Each step updates the online network by gradient descent and the target network by EMA.

\xi \leftarrow \tau \xi + (1-\tau)\theta

\theta \leftarrow \mathrm{optimizer}\!\left(\theta, \nabla_{\theta} \mathcal{L}^{\mathrm{BYOL}}_{\theta,\xi}, \eta\right)

Training Procedure

Encoder: ResNet-50 v1 by default; representation dim $2048$ .
Projector: MLP, hidden size $4096$ , output size $256$ .
Predictor: MLP on top of projector output.
Optimizer: LARS.
Epochs: $1000$ .
Batch size: $4096$ .
Learning rate: $0.2 \times \mathrm{BatchSize}/256$ with cosine decay.
Warm-up: $10$ epochs.
Weight decay: $1.5 \times 10^{-6}$ .
Target EMA base: $\tau_{\mathrm{base}} = 4 \times 10^{-3}$ .

Evaluation

Datasets

ImageNet ILSVRC-2012.
ImageNet semi-supervised splits: $1\%$ and $10\%$ labels.
Places365.
VOC2012 segmentation.
Object detection and depth-estimation transfer benchmarks.

Metrics

ImageNet top-1, top-5 linear-eval accuracy.
Semi-supervised top-1, top-5 accuracy.
Transfer classification accuracy.
Segmentation mIOU.
Detection AP metrics.
Depth metrics including RMSE and related error measures.

Headline results

ImageNet linear eval, ResNet-50: top-1 $74.3\%$ , top-5 $91.6\%$ .
ImageNet linear eval, ResNet-200 (2 $\times$ ): top-1 $79.6\%$ .
Places365 linear eval from ImageNet pretraining: top-1 $57.1\%$ , top-5 $79.8\%$ .
VOC2012 segmentation transfer: mIOU $84.5$ .
Transfer classification from ImageNet pretraining: Food101 $77.5\%$ , CIFAR-10 $96.1\%$ .

Results plot: BYOL loses less top-1 accuracy than SimCLR as batch size shrinks and augmentations are removed.

Ablations

Batch size: BYOL stays stable from $256$ to $4096$ ; SimCLR drops much faster.
Augmentations: BYOL is more resilient when grayscale, color, blur, or all but crops are removed.
Target network: fixed random targets avoid collapse but only reach $18.8\%$ top-1.
Predictor or EMA removal: representations collapse unless the predictor is made near-optimal.

Method Strengths and Weaknesses

Strengths

Removes negative pairs without losing SOTA ImageNet linear-eval accuracy.
Strong robustness to smaller batches than SimCLR.
Stronger resilience to weaker augmentation sets.
Transfers well across classification, segmentation, and detection.

Weaknesses

Collapse prevention remains empirical; theory is incomplete.
Relies on architectural asymmetry from predictor plus EMA target.
Training is long: $1000$ epochs on ImageNet.
Uses large-batch LARS pretraining in the main setup.

Suggestions from the authors

Explain why predictor plus EMA target avoids collapse.
Clarify whether negatives are ever necessary in SSL.
Test the bootstrap mechanism in other self-supervised domains.
Study which augmentations remain essential once negatives are removed.

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Problem

Framing

Currently Used Methods

Foundational

Proposed Method

Loss / Objective

Loss / Objective

Sampling Rule / Algorithm

Training Procedure

Evaluation

Datasets

Metrics

Headline results

Ablations

Method Strengths and Weaknesses

Strengths

Weaknesses

Suggestions from the authors

Links

Prior Papers

Further Papers