Segment Anything Model (SAM) is the foundational computer vision model from Meta AI that solves the general image segmentation problem with zero-shot generalization — trained on 11 million images with 1 billion masks, SAM can segment virtually any object in any image when prompted with a point, bounding box, or text, without task-specific retraining.

What Is SAM?

• Definition: A promptable segmentation foundation model that generates high-quality object masks from minimal user input — points, bounding boxes, or text prompts — across arbitrary image domains.
• Zero-Shot: Unlike previous segmentation models trained on specific object categories ("cars", "dogs"), SAM learned the generalized concept of "an object" and segments novel categories it never encountered during training.
• Scale: Trained on SA-1B dataset — 11 million images, 1 billion+ masks, the largest segmentation dataset ever assembled — by a team of annotators using SAM itself in a human-in-the-loop pipeline.
• Release: April 2023, Meta AI Research — immediate adoption across medical imaging, robotics, satellite analysis, and creative tools.

Why SAM Matters

• Eliminates Labeling Bottleneck: SAM-assisted annotation reduces mask creation from 30+ minutes to seconds per image — transforming how computer vision datasets are built.
• Universal Applicability: A single model works on medical scans, satellite imagery, microscopy, product photos, and natural scenes without fine-tuning or domain-specific retraining.
• Foundation for Specialized Models: SAM serves as a perception backbone — downstream models use SAM masks as input for tracking, 3D reconstruction, and editing.
• Interactive Editing: Users can interactively segment objects with one click rather than pixel-level manual annotation.
• Research Catalyst: SAM democratizes segmentation research — researchers apply it to new domains without requiring large labeled datasets.

Architecture

Image Encoder:

• Vision Transformer (ViT-H) encoder with 636M parameters — the computationally expensive component run once per image.
• Produces a dense image embedding (64×64 spatial resolution, 256-channel feature maps) capturing rich visual features.
• Pre-computed image embeddings enable multiple prompts on the same image without re-encoding.

Prompt Encoder:

• Lightweight encoder for prompt inputs — points (foreground/background), bounding boxes, and optionally text (via CLIP embeddings).
• Points encoded as positional embeddings + learned foreground/background tokens.
• Bounding boxes as corner point embeddings.

Mask Decoder:

• Transformer-based lightweight decoder taking image embeddings + prompt embeddings → binary segmentation masks.
• Runs in ~50ms CPU — fast enough for interactive use after one-time image encoding.
• Predicts 3 candidate masks (for ambiguous prompts) with confidence scores; user or system selects best.

Promptable Segmentation Modes

Point Prompts:

• Single click on an object → SAM segments the most likely object at that location.
• Multiple positive points refine the mask; negative points (background clicks) exclude regions.

Bounding Box Prompts:

• Draw a loose bounding box around a region → SAM segments all objects within the box.
• Useful for automating segmentation in detection pipelines.

Automatic Mask Generation:

• Run SAM with a grid of points across the entire image → generates masks for every visible object automatically.
• Used for creating comprehensive scene annotations and dataset construction.

SAM 2 (August 2024):

• Extends SAM to video — tracks and segments objects across video frames with the same prompting interface.
• Unified architecture for both image and video segmentation.
• Streaming memory mechanism maintains object identity across frames.
• 6x faster than SAM 1 on images; real-time video segmentation capability.

Applications by Domain

Limitations

• No Semantic Understanding: SAM identifies object boundaries but not what the object is — it produces masks, not class labels. Requires downstream classification.
• Small/Thin Objects: Struggles with very thin structures (wires, poles) and tiny objects below effective resolution.
• Transparent Objects: Difficulty with glass, water, and transparent materials due to ambiguous boundaries.

SAM is the "BERT moment" for image segmentation — just as BERT transformed NLP by providing a universal language understanding foundation, SAM provides a universal visual grounding foundation that every specialized segmentation and perception system can build upon.