image segmentation semantic

**Image Segmentation** is the **pixel-level computer vision task that assigns a class label (semantic), instance identity (instance), or both (panoptic) to every pixel in an image — providing the finest-grained spatial understanding of visual scenes, essential for autonomous driving, medical imaging, robotics, and any application requiring precise delineation of object boundaries rather than just bounding boxes**. **Segmentation Taxonomy** - **Semantic Segmentation**: Every pixel gets a class label (road, car, pedestrian, sky). Does not distinguish between individual instances — all cars are labeled "car". - **Instance Segmentation**: Detects individual objects and produces a binary mask for each. Distinguishes car_1 from car_2 but does not label background pixels. - **Panoptic Segmentation**: Combines both — every pixel gets a class and instance ID. "Stuff" classes (sky, road) get semantic labels; "thing" classes (car, person) get both semantic and instance labels. **Key Architectures** - **FCN (Fully Convolutional Networks)**: The foundational approach — replace FC layers with convolutions, producing a dense output map. Upsampling (transposed convolutions or bilinear) restores spatial resolution. Skip connections from encoder to decoder preserve fine spatial detail. - **U-Net**: Symmetric encoder-decoder with skip connections at every resolution level. The encoder contracts spatial dimensions while increasing feature richness; the decoder expands back. Skip connections concatenate encoder features with decoder features, preserving boundary precision. The dominant architecture for medical image segmentation. - **DeepLab v3+**: Uses atrous (dilated) convolutions to maintain large receptive fields without reducing spatial resolution. Atrous Spatial Pyramid Pooling (ASPP) captures multi-scale context by applying parallel dilated convolutions at different rates. - **Mask R-CNN**: Extends Faster R-CNN with a parallel mask prediction branch. For each detected instance, a small FCN predicts a 28×28 binary mask. The industry standard for instance segmentation. **Segment Anything Model (SAM)** Meta's foundation model for segmentation (2023): - **Image Encoder**: ViT-H processes the image once into embeddings. - **Prompt Encoder**: Accepts points, boxes, masks, or text as segmentation prompts. - **Mask Decoder**: Lightweight Transformer that produces valid masks for any prompt in real-time (~50 ms per prompt, image encoding amortized). - **Training Data**: SA-1B dataset — 1 billion masks on 11 million images, created through a data engine where SAM assisted human annotators. - **Zero-Shot Transfer**: Segments any object in any image without training on that object class, changing segmentation from a closed-vocabulary to an open-vocabulary capability. **Loss Functions** - **Cross-Entropy**: Per-pixel classification loss. Simple but treats all pixels equally, struggling with class imbalance. - **Dice Loss**: Directly optimizes the Dice coefficient (2×|A∩B|/(|A|+|B|)). Better for imbalanced classes (small objects in large images). - **Boundary Loss**: Penalizes predictions based on distance to the ground-truth boundary. Improves contour precision for medical imaging. Image Segmentation is **the pixel-level perception capability that transforms raw images into structured spatial understanding** — bridging the gap between recognizing that objects exist and knowing exactly where every part of every object is located in the scene.