Semantic Segmentation Fcn Unet Pixel-Wise Classification

# Semantic Segmentation: FCN, UNet & Pixel-Wise Classification

## Introduction & Motivation

Semantic segmentation: classify every pixel. FCN: end-to-end fully convolutional. UNet: encoder-decoder with skip connections. Applications: medical imaging, autonomous driving, scene understanding.

Motivation: Classification per-pixel; dense prediction task. Skip connections preserve spatial detail (UNet). Encoder-decoder balances efficiency and detail preservation.

Applications: Medical image segmentation, autonomous driving, satellite imagery, scene parsing.

---

## Core Concepts & Theory

### Encoder-Decoder Architecture

Encoder: downsample, extract features. Decoder: upsample, spatially recover.

### Skip Connections

Concatenate encoder features with decoder; preserve spatial detail.

### Upsampling Strategies

Bilinear interpolation, transposed convolution, dilated convolution.

---

## Mathematical Formulation

FCN output:
$$ ext{score}(x, y, c) = ext{decoder}( ext{encoder}(I))_c$$

UNet skip connection:
$$F_d = ext{upsample}(F_d) \oplus ext{encoder\_feature}$$

Segmentation loss (per-pixel cross-entropy):
$$\mathcal{L} = -\sum_i \sum_c y_{i,c} \log(\hat{y}_{i,c})$$

---

## Advanced Theory & Extensions

### Dilated Convolution

Increase receptive field without downsampling; DeepLab uses it.

### Atrous Spatial Pyramid Pooling (ASPP)

Multi-scale context; concatenate dilated convolutions.

### Conditional Random Fields (CRF)

Post-processing; enforce spatial smoothness.

---

## Computational Considerations

FCN: O(H × W × C) per pixel; fully convolutional, efficient.

UNet: O(H × W × C) encoding + decoding.

CRF refinement: O(H × W) pairwise terms; iterative optimization.

---

## Practical Implementation Strategies

### Input Resolution

Balance memory and detail; typical 256-512 for medical, 512-1024 for driving.

### Loss Weighting

Weight rare classes higher; address class imbalance.

### Test-Time Augmentation

Predict on multiple crops; average predictions.

---

## Benchmark Datasets & Evaluation

Cityscapes: Autonomous driving; 19 classes, 2048×1024.

Pascal VOC: Scene segmentation; 21 classes.

Medical Imaging: Organ/tissue segmentation; class-specific metrics.

Metrics: mIoU (mean Intersection over Union), per-class IoU.

---

## Key Challenges & Limitations

### Class Imbalance

Background dominates; rare classes underrepresented.

### Boundary Accuracy

Sharp boundaries hard to capture; CRF post-processing helps.

### Computational Cost

Dense prediction expensive; downsampling necessary.

---

## Hyperparameter Tuning

Encoder depth: ResNet-50/101; balance accuracy-speed.

Decoder channels: 256-512; skip connection preservation.

Dilation rates (ASPP): [1, 6, 12, 18]; multi-scale context.

---

## Real-World Applications & Case Studies

Medical Imaging: UNet standard for organ/tumor segmentation.

Autonomous Driving: Cityscapes-trained models for scene understanding.

Satellite Imagery: Land-use classification via segmentation.

---

## Integration with Other Methods

Segmentation + Instance Segmentation → per-instance masks.

Segmentation + Boundary Detection → refine edges.

---

## Summary & Key Takeaways

Semantic segmentation via FCN/UNet enables pixel-wise dense prediction through encoder-decoder architecture with skip connections, achieving accurate spatial localization.

Principles:
1. Encoder-decoder: downsampling then upsampling.
2. Skip connections: preserve spatial detail from encoder.
3. Dilated convolution: increase receptive field.
4. Per-pixel cross-entropy loss; weight rare classes.
5. CRF post-processing enforces spatial smoothness.

---

---

## Appendix: Practical Labs

### Lab 1: Upsampling Strategies

import torch
import torch.nn.functional as F
import numpy as np

def upsample_bilinear(x, scale_factor=2):
 """Bilinear upsampling"""
 return F.interpolate(x, scale_factor=scale_factor, mode='bilinear', align_corners=False)

def upsample_transposed_conv(x, in_channels, out_channels, scale_factor=2):
 """Transposed convolution (learnable upsampling)"""
 kernel_size = 2 * scale_factor
 stride = scale_factor
 conv = torch.nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride)
 return conv(x)

# Test
x = torch.randn(2, 64, 16, 16)
x_up_bilinear = upsample_bilinear(x, scale_factor=2)

print(f"Original shape: {x.shape}, Upsampled: {x_up_bilinear.shape}")
assert x_up_bilinear.shape == (2, 64, 32, 32), "Should upsampled correctly"
print("✓ Upsampling working")

if __name__ == "__main__":
 print("Lab 1: Upsampling - PASSED")

### Lab 2: Skip Connection Concatenation

import torch
import torch.nn as nn
import torch.nn.functional as F

def skip_connection(encoder_feature, decoder_feature):
 """Concatenate encoder and decoder features"""
 # Ensure same spatial resolution
 if encoder_feature.shape[-2:] != decoder_feature.shape[-2:]:
 decoder_feature = F.interpolate(decoder_feature, size=encoder_feature.shape[-2:], mode='bilinear', align_corners=False)
 
 concatenated = torch.cat([encoder_feature, decoder_feature], dim=1)
 return concatenated

# Test
encoder_feat = torch.randn(2, 256, 32, 32)
decoder_feat = torch.randn(2, 128, 32, 32)

skip_out = skip_connection(encoder_feat, decoder_feat)

print(f"Skip output shape: {skip_out.shape}")
assert skip_out.shape == (2, 384, 32, 32), "Should concatenate on channel dim"
print("✓ Skip connection working")

if __name__ == "__main__":
 print("Lab 2: Skip Connection - PASSED")

### Lab 3: Per-Pixel Cross-Entropy Loss

import torch
import torch.nn as nn
import numpy as np

def compute_segmentation_loss(logits, targets, class_weights=None):
 """Per-pixel cross-entropy loss"""
 # logits: [B, C, H, W]
 # targets: [B, H, W] (class indices)
 
 criterion = nn.CrossEntropyLoss(weight=class_weights)
 loss = criterion(logits, targets)
 return loss

# Test
batch_size, n_classes, height, width = 4, 5, 32, 32
logits = torch.randn(batch_size, n_classes, height, width)
targets = torch.randint(0, n_classes, (batch_size, height, width))

loss = compute_segmentation_loss(logits, targets)

print(f"Segmentation loss: {loss:.4f}")
assert loss > 0, "Loss should be positive"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Segmentation loss working")

if __name__ == "__main__":
 print("Lab 3: Segmentation Loss - PASSED")

### Lab 4: Segmentation Metrics

import torch
import numpy as np

def compute_iou_per_class(predictions, targets, n_classes):
 """Compute IoU for each class; average for mIoU"""
 ious = []
 
 for c in range(n_classes):
 pred_mask = (predictions == c)
 target_mask = (targets == c)
 
 intersection = (pred_mask & target_mask).sum()
 union = (pred_mask | target_mask).sum()
 
 iou = intersection / (union + 1e-8)
 ious.append(iou.item())
 
 return np.array(ious)

# Test
predictions = torch.randint(0, 5, (4, 32, 32))
targets = torch.randint(0, 5, (4, 32, 32))

ious = compute_iou_per_class(predictions, targets, n_classes=5)

print(f"IoU per class: {ious}")
print(f"mIoU: {ious.mean():.4f}")
assert len(ious) == 5, "Should have 5 IoUs"
assert all(0 <= iou <= 1 for iou in ious), "IoU should be in [0,1]"
print("✓ Segmentation metrics working")

if __name__ == "__main__":
 print("Lab 4: Metrics - PASSED")