semantic segmentation

**Semantic Segmentation** is the **computer vision task of assigning a class label to every pixel in an image** — enabling precise understanding of scene composition at pixel level, essential for autonomous driving, medical imaging, and satellite analysis. **Segmentation Types** - **Semantic**: Each pixel gets a class (car, road, sky) — no instance distinction. - **Instance**: Each object instance labeled separately — two cars = two labels. - **Panoptic**: Semantic + instance for all pixels simultaneously — unified framework. **U-Net Architecture (2015)** - Encoder-decoder with skip connections. - **Encoder**: Contracting path — conv + pool, captures context at multiple scales. - **Decoder**: Expanding path — upsample + conv, recovers spatial resolution. - **Skip connections**: Concatenate encoder feature maps to decoder — preserve fine details. - Originally for biomedical segmentation; now universal. - U-Net variants: U-Net++, Attention U-Net, nnU-Net (self-configuring), Swin U-Net. **DeepLab Series** - DeepLab v3+: Atrous (dilated) convolutions — expand receptive field without losing resolution. - Atrous Spatial Pyramid Pooling (ASPP): Multi-scale context aggregation. - Achieves 89.0% mIoU on Cityscapes. **SegFormer (2021)** - Hierarchical Transformer encoder (Mix Transformer, MiT). - Lightweight MLP decoder — simple but effective. - No positional encoding → generalizes to different image resolutions. - SegFormer-B5: 84.0 mIoU on ADE20K — efficient and accurate. **SAM (Segment Anything Model, 2023)** - Foundation model for segmentation. - Prompt-based: Point, box, or text prompt → segment any object. - Trained on 1.1B masks — largest segmentation dataset. - SAM2: Extends to video segmentation. **Evaluation Metrics** - **mIoU (mean Intersection over Union)**: Average IoU across all classes. - **Pixel Accuracy**: Fraction of correctly classified pixels. Semantic segmentation is **the key enabler of pixel-level scene understanding** — from autonomous vehicle perception stacks requiring lane/obstacle delineation to medical AI systems detecting tumor boundaries, dense prediction at pixel resolution transforms raw images into actionable spatial intelligence.