sam

**Segment Anything Model (SAM)** is the **foundational computer vision model from Meta AI that solves the general image segmentation problem with zero-shot generalization** — trained on 11 million images with 1 billion masks, SAM can segment virtually any object in any image when prompted with a point, bounding box, or text, without task-specific retraining. **What Is SAM?** - **Definition**: A promptable segmentation foundation model that generates high-quality object masks from minimal user input — points, bounding boxes, or text prompts — across arbitrary image domains. - **Zero-Shot**: Unlike previous segmentation models trained on specific object categories ("cars", "dogs"), SAM learned the generalized concept of "an object" and segments novel categories it never encountered during training. - **Scale**: Trained on SA-1B dataset — 11 million images, 1 billion+ masks, the largest segmentation dataset ever assembled — by a team of annotators using SAM itself in a human-in-the-loop pipeline. - **Release**: April 2023, Meta AI Research — immediate adoption across medical imaging, robotics, satellite analysis, and creative tools. **Why SAM Matters** - **Eliminates Labeling Bottleneck**: SAM-assisted annotation reduces mask creation from 30+ minutes to seconds per image — transforming how computer vision datasets are built. - **Universal Applicability**: A single model works on medical scans, satellite imagery, microscopy, product photos, and natural scenes without fine-tuning or domain-specific retraining. - **Foundation for Specialized Models**: SAM serves as a perception backbone — downstream models use SAM masks as input for tracking, 3D reconstruction, and editing. - **Interactive Editing**: Users can interactively segment objects with one click rather than pixel-level manual annotation. - **Research Catalyst**: SAM democratizes segmentation research — researchers apply it to new domains without requiring large labeled datasets. **Architecture** **Image Encoder**: - Vision Transformer (ViT-H) encoder with 636M parameters — the computationally expensive component run once per image. - Produces a dense image embedding (64×64 spatial resolution, 256-channel feature maps) capturing rich visual features. - Pre-computed image embeddings enable multiple prompts on the same image without re-encoding. **Prompt Encoder**: - Lightweight encoder for prompt inputs — points (foreground/background), bounding boxes, and optionally text (via CLIP embeddings). - Points encoded as positional embeddings + learned foreground/background tokens. - Bounding boxes as corner point embeddings. **Mask Decoder**: - Transformer-based lightweight decoder taking image embeddings + prompt embeddings → binary segmentation masks. - Runs in ~50ms CPU — fast enough for interactive use after one-time image encoding. - Predicts 3 candidate masks (for ambiguous prompts) with confidence scores; user or system selects best. **Promptable Segmentation Modes** **Point Prompts**: - Single click on an object → SAM segments the most likely object at that location. - Multiple positive points refine the mask; negative points (background clicks) exclude regions. **Bounding Box Prompts**: - Draw a loose bounding box around a region → SAM segments all objects within the box. - Useful for automating segmentation in detection pipelines. **Automatic Mask Generation**: - Run SAM with a grid of points across the entire image → generates masks for every visible object automatically. - Used for creating comprehensive scene annotations and dataset construction. **SAM 2 (August 2024)**: - Extends SAM to video — tracks and segments objects across video frames with the same prompting interface. - Unified architecture for both image and video segmentation. - Streaming memory mechanism maintains object identity across frames. - 6x faster than SAM 1 on images; real-time video segmentation capability. **Applications by Domain** | Domain | Application | SAM Benefit | |--------|-------------|-------------| | Medical imaging | Tumor boundary delineation | Clicks replace 30-min manual tracing | | Robotics | Object localization for grasping | Zero-shot across new object categories | | Satellite | Land cover mapping | Segment fields, buildings, roads universally | | Creative tools | Background removal | One-click subject isolation | | AR/VR | Scene decomposition | Real-time object separation | | Dataset creation | Annotation acceleration | 10-100x speedup over manual polygon tools | **Limitations** - **No Semantic Understanding**: SAM identifies object boundaries but not what the object is — it produces masks, not class labels. Requires downstream classification. - **Small/Thin Objects**: Struggles with very thin structures (wires, poles) and tiny objects below effective resolution. - **Transparent Objects**: Difficulty with glass, water, and transparent materials due to ambiguous boundaries. SAM is **the "BERT moment" for image segmentation** — just as BERT transformed NLP by providing a universal language understanding foundation, SAM provides a universal visual grounding foundation that every specialized segmentation and perception system can build upon.