semantic segmentation deep learning

**Semantic Segmentation** is **the dense prediction task that assigns a class label to every pixel in an image — requiring the network to simultaneously understand global scene context (what objects are present) and precise local boundaries (exactly where each object begins and ends), producing a pixel-wise classification map matching the input resolution**. **Foundational Architectures:** - **Fully Convolutional Network (FCN)**: replaced final FC layers with 1×1 convolutions and added transposed convolutions for upsampling — first end-to-end trainable segmentation network; skip connections from earlier layers recover spatial detail lost during downsampling - **U-Net**: symmetric encoder-decoder with dense skip connections at every resolution level — encoder captures context through downsampling, decoder reconstructs spatial detail through upsampling; skip connections concatenate encoder features to decoder for precise boundary recovery; dominant in medical image segmentation - **DeepLab v3+**: atrous (dilated) convolutions with Atrous Spatial Pyramid Pooling (ASPP) — dilated convolutions expand receptive field without reducing spatial resolution; ASPP applies parallel dilated convolutions at multiple rates to capture multi-scale context - **PSPNet**: Pyramid Pooling Module aggregates features at multiple scales — global average pooling, 6×6, 3×3, and 1×1 pooling branches capture sub-region patterns at different granularities; outperforms FCN on complex scenes **Modern Approaches:** - **Transformer-Based (SegFormer, Mask2Former)**: self-attention captures global context from the first layer — hierarchical Transformer encoders produce multi-scale features; MLP decoders aggregate features from all scales for segmentation prediction - **Panoptic Segmentation**: unified framework combining semantic segmentation (stuff: sky, road) with instance segmentation (things: cars, people) — Panoptic FPN, MaskFormer, and Mask2Former produce both class-labeled regions and individual instance masks - **Real-Time Segmentation**: BiSeNet, DDRNet use dual-branch architectures — one branch for spatial detail (high resolution, shallow), one for semantic context (low resolution, deep); achieves >30 FPS at reasonable accuracy for autonomous driving - **Semi-Supervised Methods**: leverage unlabeled images with pseudo-labels generated by teacher model — FixMatch-based approaches achieve near-supervised accuracy using only 1-10% labeled data **Training and Evaluation:** - **Loss Functions**: cross-entropy per pixel is baseline — Dice loss handles class imbalance by optimizing the overlap ratio directly; Lovász loss is a smooth surrogate for IoU optimization; combination losses (CE + Dice) common in practice - **Mean IoU (mIoU)**: primary evaluation metric — IoU per class = intersection/union of predicted and ground-truth pixels; averaged across all classes; IoU penalizes both false positives and false negatives equally - **Data Augmentation**: random scaling (0.5-2.0×), random crop, horizontal flip, color jitter — multi-scale training helps the model generalize across object sizes - **Class Imbalance**: outdoor scenes dominated by sky, road, vegetation — class-weighted loss, oversampling rare classes, or Online Hard Example Mining (OHEM) focuses training on under-represented or difficult pixels **Semantic segmentation is the visual understanding task that enables pixel-level scene parsing — powering autonomous driving perception, medical image analysis, satellite imagery interpretation, and augmented reality by providing the dense spatial understanding that detection and classification alone cannot achieve.**