Masked Autoencoders Are Scalable Vision Learners

Problem

Framing

Vision self-supervision had not matched supervised ViT pre-training at scale, and masked prediction looked too easy on redundant images. MAE closes this gap with heavy patch masking and an asymmetric encoder-decoder that reconstructs pixels from only visible patches. On ImageNet-1K, ViT-H reaches 87.8% top-1.

Currently Used Methods

Foundational

• @chenSimCLR2020 — contrastive pre-training with augmented view agreement.
  • Limitation in context: needs strong augmentations and still trails MAE in transfer.
• @grillBYOL2020 — non-contrastive bootstrap self-distillation without negatives.
  • Limitation in context: learns invariances, not explicit missing-content prediction.
• @caronDINO2021 — self-distilled ViT features with strong linear separability.
  • Limitation in context: weaker end-to-end transfer than MAE at larger scales.
• @dosovitskiyViT2020 — pure transformer backbone for image patches.
  • Limitation in context: supervised pre-training demands labels and large compute.

Proposed Method

Architecture

MAE splits an image into patches, masks about 75%, and feeds only visible patches to the ViT encoder. Mask tokens are inserted only after the encoder, then a small decoder reconstructs all patches; the decoder is discarded for downstream use. The default decoder has depth 8 and width 512.

Loss / Objective

The training target is pixel reconstruction on masked patches only.

Algorithm

Pre-training samples a random mask set, encodes visible patches, decodes the full sequence, and optimizes masked-patch MSE.

Training Procedure

• Masking ratio: 75% default.
• Pre-training length: 800 epochs default; 1600 for strongest models.
• Optimizer: AdamW.
• Base learning rate: $1.5 \times 10^{-4}$.
• Weight decay: 0.05.
• Decoder depth: 8.
• Decoder width: 512.
• Data augmentation: random resized crop.

Evaluation

Datasets

• ImageNet-1K pre-training and validation.
• COCO for detection and instance segmentation transfer.
• ADE20K for semantic segmentation transfer.
• iNaturalist 2017/2018/2019 for classification transfer.
• Places205 for scene classification transfer.

Metrics

• ImageNet top-1 accuracy.
• Linear probing accuracy.
• COCO $\mathrm{AP}^{\mathrm{box}}$.
• COCO $\mathrm{AP}^{\mathrm{mask}}$.
• ADE20K mIoU.

Headline results

• ImageNet-1K fine-tuning (ViT-H): 87.8% top-1.
• ImageNet-1K fine-tuning (ViT-L, 1600 epochs): 85.9% top-1.
• COCO detection (ViT-L): $49.9\;\mathrm{AP}^{\mathrm{box}}$.
• COCO instance segmentation (ViT-L): $44.3\;\mathrm{AP}^{\mathrm{mask}}$.
• iNat2018 transfer (ViT-H448): 86.8% top-1.

Ablations

• Masking ratio: 75% is best; low masking under-trains the representation.
• Decoder depth: deeper helps linear probing more than fine-tuning.
• Decoder width: larger width improves linear probing, small effect on fine-tuning.
• Mask tokens in encoder: removing them cuts encoder FLOPs by about $3.3\times$.

Method Strengths and Weaknesses

Strengths

• High masking makes prediction nontrivial and improves accuracy.
• Asymmetric design cuts pre-training cost by $3\times$ or more.
• Scaling is strong: ViT-H reaches 87.8% with ImageNet-1K only.
• Transfer beats supervised pre-training on detection and segmentation.

Weaknesses

• Linear probing is weaker than DINO-style features.
• Pixel reconstruction target may under-emphasize semantic abstraction.
• Best accuracy needs long pre-training up to 1600 epochs.
• Method depends on ViT patch tokenization and large-model capacity.

Suggestions from the authors

• Study why pixel reconstruction yields strong transfer despite weak linear probing.
• Scale masked autoencoding to larger models and longer training.
• Explore reconstruction targets beyond raw pixels.
• Extend masked autoencoding to broader vision tasks and data regimes.

Links

Prior Papers

• @caronDINO2021 — DINO is a strong contemporary ViT self-supervision baseline that MAE compares against.
• @chenSimCLR2020 — SimCLR anchors the contrastive-learning baseline family that MAE departs from.
• @grillBYOL2020 — BYOL is a key non-contrastive baseline for representation learning before MAE.
• @dosovitskiyViT2020 — ViT provides the transformer backbone that makes masked patch modeling practical.

Further Papers

• @kirillovSAM2023 — SAM uses MAE-pretrained ViT-style encoders in large-scale vision foundation modeling.
• @liuSwinTransformer2021 — Swin is a later overlapping vision-transformer direction for scalable representation learning.
• @peeblesDiT2022 — DiT extends transformer-based image modeling and inherits the ViT scaling context MAE helped normalize.