Momentum Contrast for Unsupervised Visual Representation Learning

Problem

Framing

Contrastive visual pretraining faced a trade-off: end-to-end dictionaries were small, while memory banks were stale. MoCo closes it with a queue-based dictionary and a momentum key encoder that keeps many negatives usable without exploding batch size. On ImageNet linear evaluation, ResNet-50 reaches 60.6% top-1.

Currently Used Methods

Foundational

• @heDeepResidualLearning2016 — ResNet supplies the backbone used for all MoCo encoders.
  • Limitation in context: architecture alone gives no unsupervised dictionary-learning mechanism.
• @chenSimCLR2020 — large-batch end-to-end contrastive learning with in-batch negatives.
  • Limitation in context: dictionary size scales with batch size, raising memory cost sharply.
• "Unsupervised Feature Learning via Non-Parametric Instance Discrimination" — memory-bank contrastive learning with many stored negatives.
  • Limitation in context: stored keys become stale as the encoder changes.
• "Contrastive Predictive Coding" — InfoNCE-style contrastive prediction for representation learning.
  • Limitation in context: does not solve large, consistent negative dictionaries for vision pretraining.

Proposed Method

Architecture

MoCo uses two encoders with identical architecture: a query encoder $f_q$ and a key encoder $f_k$. The current mini-batch produces queries and positive keys, while older keys remain in a FIFO queue, so dictionary size $K$ is decoupled from batch size.

Loss / Objective

The objective is InfoNCE over one positive key and $K$ dictionary negatives.

Sampling Rule / Algorithm

The key encoder evolves by exponential moving average of the query encoder.

Training Procedure

• Backbone: ResNet-50.
• Batch size: 256 on 8 GPUs.
• Epochs: 200 on ImageNet-1M.
• Optimizer: SGD, lr $0.03$.
• Weight decay: $10^{-4}$.
• SGD momentum: $0.9$.
• LR drops: at epochs 120 and 160 by $0.1\times$.
• Queue size: $K=65536$.
• Momentum coefficient: $m=0.999$.
• Temperature: $T=0.07$.

Evaluation

Datasets

• ImageNet-1M linear classification.
• Instagram-1B pretraining transfer.
• PASCAL VOC detection and segmentation.
• COCO detection, instance segmentation, and keypoint detection.

Metrics

• ImageNet: top-1 linear classification accuracy.
• Detection: $\mathrm{AP}^{\mathrm{bb}}$, $\mathrm{AP}_{50}^{\mathrm{bb}}$, $\mathrm{AP}_{75}^{\mathrm{bb}}$.
• Instance segmentation: $\mathrm{AP}^{\mathrm{mk}}$, $\mathrm{AP}_{50}^{\mathrm{mk}}$, $\mathrm{AP}_{75}^{\mathrm{mk}}$.
• Keypoints: $\mathrm{AP}^{\mathrm{kp}}$, $\mathrm{AP}_{50}^{\mathrm{kp}}$, $\mathrm{AP}_{75}^{\mathrm{kp}}$.

Headline results

• ImageNet linear, ResNet-50, 200 epochs: 60.6% top-1.
• ImageNet linear, larger queue $K=16384$: 60.4% top-1.
• VOC/COCO transfer: MoCo beats supervised pretraining on 7 detection/segmentation tasks.
• COCO keypoints: MoCo remains slightly below supervised pretraining.

Table 3: Scaling ImageNet linear accuracy against model size, compared with prior unsupervised methods.

Ablations

• Dictionary size $K$: larger queues improve accuracy, saturating near $K=65536$.
• Contrastive mechanism: MoCo beats memory bank and end-to-end at matched dictionary size.
• Momentum encoder: slow key updates stabilize a large queue.
• Model width: wider ResNets improve linear accuracy, reaching 68.6% with R50w4x.

Method Strengths and Weaknesses

Strengths

• Decouples negative-set size from batch size through a FIFO queue.
• Momentum encoder reduces representation drift in stored keys.
• Reaches 60.6% ImageNet linear accuracy with standard ResNet-50.
• Transfer beats supervised pretraining on seven detection and segmentation tasks.

Weaknesses

• Still trails strongest large-model unsupervised baselines in some linear-probe comparisons.
• Depends on a hand-set momentum coefficient and queue size.
• Uses a separate key encoder, adding state and implementation complexity.
• COCO keypoint detection does not surpass better supervised pretraining.

Suggestions from the authors

• Apply MoCo to pretext tasks beyond instance discrimination.
• Study larger-scale pretraining on web-scale image collections.
• Improve unsupervised features for tasks where supervision still leads.
• Analyze stronger object-detection fine-tuning schedules and backbones.

Links

Prior Papers

• @heDeepResidualLearning2016 — ResNet is the encoder backbone MoCo instantiates for contrastive pretraining.

Further Papers

• @chenSimCLR2020 — SimCLR is the direct large-batch contrastive alternative MoCo is commonly compared against.
• @grillBYOL2020 — BYOL keeps the momentum-encoder idea but removes explicit negative pairs.