Semantic Segmentation is the computer vision task of assigning a class label to every pixel in an image — enabling precise understanding of scene composition at pixel level, essential for autonomous driving, medical imaging, and satellite analysis.

Segmentation Types

• Semantic: Each pixel gets a class (car, road, sky) — no instance distinction.
• Instance: Each object instance labeled separately — two cars = two labels.
• Panoptic: Semantic + instance for all pixels simultaneously — unified framework.

U-Net Architecture (2015)

• Encoder-decoder with skip connections.
• Encoder: Contracting path — conv + pool, captures context at multiple scales.
• Decoder: Expanding path — upsample + conv, recovers spatial resolution.
• Skip connections: Concatenate encoder feature maps to decoder — preserve fine details.
• Originally for biomedical segmentation; now universal.
• U-Net variants: U-Net++, Attention U-Net, nnU-Net (self-configuring), Swin U-Net.

DeepLab Series

• DeepLab v3+: Atrous (dilated) convolutions — expand receptive field without losing resolution.
• Atrous Spatial Pyramid Pooling (ASPP): Multi-scale context aggregation.
• Achieves 89.0% mIoU on Cityscapes.

SegFormer (2021)

• Hierarchical Transformer encoder (Mix Transformer, MiT).
• Lightweight MLP decoder — simple but effective.
• No positional encoding → generalizes to different image resolutions.
• SegFormer-B5: 84.0 mIoU on ADE20K — efficient and accurate.

SAM (Segment Anything Model, 2023)

• Foundation model for segmentation.
• Prompt-based: Point, box, or text prompt → segment any object.
• Trained on 1.1B masks — largest segmentation dataset.
• SAM2: Extends to video segmentation.

Evaluation Metrics

• mIoU (mean Intersection over Union): Average IoU across all classes.
• Pixel Accuracy: Fraction of correctly classified pixels.

Semantic segmentation is the key enabler of pixel-level scene understanding — from autonomous vehicle perception stacks requiring lane/obstacle delineation to medical AI systems detecting tumor boundaries, dense prediction at pixel resolution transforms raw images into actionable spatial intelligence.