Segment Anything

Problem

Framing

Open-vocabulary segmentation lacked a promptable model that transfers without task-specific fine-tuning and has data at foundation-model scale. SAM closes this with a prompt encoder plus lightweight mask decoder trained on SA-1B, a dataset of 11M images and 1.1B masks.

Currently Used Methods

Foundational

• @caronDINO2021 — self-distilled ViT pretraining yields strong transferable visual features.
  • Limitation in context: features alone do not produce promptable masks.
• @heMaskRCNN2017 — two-stage detection-segmentation for instance masks from box proposals.
  • Limitation in context: requires fixed label space and supervised downstream training.
• @dosovitskiyViT2020 — pure transformer image encoder scales representation learning.
  • Limitation in context: encoder alone lacks prompt-conditioned mask prediction.
• @heMAE2021 — masked image pretraining improves dense-transfer backbones.
  • Limitation in context: pretraining objective is not segmentation-aligned.
• @radfordCLIP2021 — language-supervised vision model enables strong zero-shot transfer.
  • Limitation in context: outputs image-text alignment, not precise masks.

Proposed Method

Architecture

SAM uses a heavyweight ViT image encoder, a prompt encoder for points, boxes, text, and masks, and a lightweight mask decoder. The image embedding is computed once, then queried by prompts to emit multiple valid masks and an IoU confidence score.

Loss / Objective

The decoder optimizes mask quality and mask ranking jointly.

Algorithm

For ambiguous prompts, SAM predicts several candidate masks and selects by predicted IoU.

Training Procedure

• Dataset: SA-1B, 11M images, 1.1B masks.
• Image encoder: scaled from ViT-B to ViT-H.
• Masks per GPU: up to 64 randomly sampled masks.
• Data mix: default training uses automatic masks only.
• Filtering: discard masks covering more than 90% of an image.

Evaluation

Datasets

• 23-dataset zero-shot segmentation suite.
• COCO.
• LVIS v1.
• BSDS500.
• ADE20K.
• Cityscapes.
• Open Images.

Metrics

• mIoU for point-prompted segmentation.
• AP for instance segmentation.
• AR@1000 for proposal quality.
• Human mask-quality rating, 1 to 10.

Headline results

• COCO zero-shot instance segmentation: AP 46.5.
• LVIS v1 zero-shot instance segmentation: AP 44.7.
• COCO ViTDet-H supervised baseline: AP 51.0.
• LVIS v1 ViTDet-H supervised baseline: AP 46.6.
• Human rating on text-to-mask: SAM 8.1, ViTDet-H 7.9.

Ablations

• Data-engine stages: each stage improves mean mIoU on the 23-dataset suite.
• Training-data scale: roughly 10% of SA-1B is close to full SA-1B.
• Image-encoder scale: ViT-H clearly beats ViT-B, but only slightly beats ViT-L.
• Automatic-only training: similar to using masks from all data-engine stages.

Method Strengths and Weaknesses

Strengths

• Unifies points, boxes, text, and masks in one promptable interface.
• Image embedding is amortized, enabling real-time prompt queries.
• Zero-shot transfer reaches 46.5 AP on COCO instance segmentation.
• SA-1B gives segmentation pretraining unprecedented mask scale.

Weaknesses

• Still trails supervised ViTDet-H on COCO and LVIS AP.
• Misses fine structures and can hallucinate small regions.
• Ambiguous prompts require multiple-mask output and reranking.
• Strong performance depends on a massive bespoke data engine.

Suggestions from the authors

• Improve fine-structure segmentation and reduce small-region hallucinations.
• Extend promptable segmentation toward richer text-conditioned outputs.
• Study representation quality from SAM for broader downstream tasks.
• Expand automatic data collection for even larger segmentation corpora.

Links

Prior Papers

• @caronDINO2021 — SAM inherits the foundation-model recipe of large-scale pretraining for transferable visual features.
• @dosovitskiyViT2020 — SAM uses a ViT family image encoder as its heavyweight visual backbone.
• @heMaskRCNN2017 — SAM targets mask prediction at object level but removes fixed-category supervision at inference.
• @radfordCLIP2021 — SAM mirrors CLIP-style zero-shot transfer goals, but for segmentation masks.

Further Papers

• @luUnifiedIO2_2024 — extends promptable perception across tasks, including segmentation-like outputs, beyond SAM's interface.
• @wuNExTGPT2023 — multimodal systems can use SAM-style segmentation as a visual grounding primitive.
• @chameleonTeam2024 — general multimodal foundation models overlap with SAM's foundation-model framing for vision.