Image Segmentation is the pixel-level computer vision task that assigns a class label (semantic), instance identity (instance), or both (panoptic) to every pixel in an image — providing the finest-grained spatial understanding of visual scenes, essential for autonomous driving, medical imaging, robotics, and any application requiring precise delineation of object boundaries rather than just bounding boxes.

Segmentation Taxonomy

• Semantic Segmentation: Every pixel gets a class label (road, car, pedestrian, sky). Does not distinguish between individual instances — all cars are labeled "car".
• Instance Segmentation: Detects individual objects and produces a binary mask for each. Distinguishes car_1 from car_2 but does not label background pixels.
• Panoptic Segmentation: Combines both — every pixel gets a class and instance ID. "Stuff" classes (sky, road) get semantic labels; "thing" classes (car, person) get both semantic and instance labels.

Key Architectures

• FCN (Fully Convolutional Networks): The foundational approach — replace FC layers with convolutions, producing a dense output map. Upsampling (transposed convolutions or bilinear) restores spatial resolution. Skip connections from encoder to decoder preserve fine spatial detail.
• U-Net: Symmetric encoder-decoder with skip connections at every resolution level. The encoder contracts spatial dimensions while increasing feature richness; the decoder expands back. Skip connections concatenate encoder features with decoder features, preserving boundary precision. The dominant architecture for medical image segmentation.
• DeepLab v3+: Uses atrous (dilated) convolutions to maintain large receptive fields without reducing spatial resolution. Atrous Spatial Pyramid Pooling (ASPP) captures multi-scale context by applying parallel dilated convolutions at different rates.
• Mask R-CNN: Extends Faster R-CNN with a parallel mask prediction branch. For each detected instance, a small FCN predicts a 28×28 binary mask. The industry standard for instance segmentation.

Segment Anything Model (SAM)

Meta's foundation model for segmentation (2023):

• Image Encoder: ViT-H processes the image once into embeddings.
• Prompt Encoder: Accepts points, boxes, masks, or text as segmentation prompts.
• Mask Decoder: Lightweight Transformer that produces valid masks for any prompt in real-time (~50 ms per prompt, image encoding amortized).
• Training Data: SA-1B dataset — 1 billion masks on 11 million images, created through a data engine where SAM assisted human annotators.
• Zero-Shot Transfer: Segments any object in any image without training on that object class, changing segmentation from a closed-vocabulary to an open-vocabulary capability.

Loss Functions

• Cross-Entropy: Per-pixel classification loss. Simple but treats all pixels equally, struggling with class imbalance.
• Dice Loss: Directly optimizes the Dice coefficient (2×|A∩B|/(|A|+|B|)). Better for imbalanced classes (small objects in large images).
• Boundary Loss: Penalizes predictions based on distance to the ground-truth boundary. Improves contour precision for medical imaging.

Image Segmentation is the pixel-level perception capability that transforms raw images into structured spatial understanding — bridging the gap between recognizing that objects exist and knowing exactly where every part of every object is located in the scene.