Semantic Segmentation Fcn Unet Pixel-Wise Classification
# Semantic Segmentation: FCN, UNet & Pixel-Wise Classification
## Introduction & Motivation
Semantic segmentation: classify every pixel. FCN: end-to-end fully convolutional. UNet: encoder-decoder with skip connections. Applications: medical imaging, autonomous driving, scene understanding.
Motivation: Classification per-pixel; dense prediction task. Skip connections preserve spatial detail (UNet). Encoder-decoder balances efficiency and detail preservation.
Applications: Medical image segmentation, autonomous driving, satellite imagery, scene parsing.
---
## Core Concepts & Theory
### Encoder-Decoder Architecture
Encoder: downsample, extract features. Decoder: upsample, spatially recover.
### Skip Connections
Concatenate encoder features with decoder; preserve spatial detail.
### Upsampling Strategies
Bilinear interpolation, transposed convolution, dilated convolution.
---
## Mathematical Formulation
FCN output:
$$ ext{score}(x, y, c) = ext{decoder}( ext{encoder}(I))_c$$
UNet skip connection:
$$F_d = ext{upsample}(F_d) \oplus ext{encoder\_feature}$$
Segmentation loss (per-pixel cross-entropy):
$$\mathcal{L} = -\sum_i \sum_c y_{i,c} \log(\hat{y}_{i,c})$$
---
## Advanced Theory & Extensions
### Dilated Convolution
Increase receptive field without downsampling; DeepLab uses it.
### Atrous Spatial Pyramid Pooling (ASPP)
Multi-scale context; concatenate dilated convolutions.
### Conditional Random Fields (CRF)
Post-processing; enforce spatial smoothness.
---
## Computational Considerations
FCN: O(H × W × C) per pixel; fully convolutional, efficient.
UNet: O(H × W × C) encoding + decoding.
CRF refinement: O(H × W) pairwise terms; iterative optimization.
---
## Practical Implementation Strategies
### Input Resolution
Balance memory and detail; typical 256-512 for medical, 512-1024 for driving.
### Loss Weighting
Weight rare classes higher; address class imbalance.
### Test-Time Augmentation
Predict on multiple crops; average predictions.
---
## Benchmark Datasets & Evaluation
Cityscapes: Autonomous driving; 19 classes, 2048×1024.
Pascal VOC: Scene segmentation; 21 classes.
Medical Imaging: Organ/tissue segmentation; class-specific metrics.
Metrics: mIoU (mean Intersection over Union), per-class IoU.
---
## Key Challenges & Limitations
### Class Imbalance
Background dominates; rare classes underrepresented.
### Boundary Accuracy
Sharp boundaries hard to capture; CRF post-processing helps.
### Computational Cost
Dense prediction expensive; downsampling necessary.
---
## Hyperparameter Tuning
Encoder depth: ResNet-50/101; balance accuracy-speed.
Decoder channels: 256-512; skip connection preservation.
Dilation rates (ASPP): [1, 6, 12, 18]; multi-scale context.
---
## Real-World Applications & Case Studies
Medical Imaging: UNet standard for organ/tumor segmentation.
Autonomous Driving: Cityscapes-trained models for scene understanding.
Satellite Imagery: Land-use classification via segmentation.
---
## Integration with Other Methods
Segmentation + Instance Segmentation → per-instance masks.
Segmentation + Boundary Detection → refine edges.
---
## Summary & Key Takeaways
Semantic segmentation via FCN/UNet enables pixel-wise dense prediction through encoder-decoder architecture with skip connections, achieving accurate spatial localization.
Principles:
1. Encoder-decoder: downsampling then upsampling.
2. Skip connections: preserve spatial detail from encoder.
3. Dilated convolution: increase receptive field.
4. Per-pixel cross-entropy loss; weight rare classes.
5. CRF post-processing enforces spatial smoothness.
---
---
## Appendix: Practical Labs
### Lab 1: Upsampling Strategies
import torch
import torch.nn.functional as F
import numpy as np
def upsample_bilinear(x, scale_factor=2):
"""Bilinear upsampling"""
return F.interpolate(x, scale_factor=scale_factor, mode='bilinear', align_corners=False)
def upsample_transposed_conv(x, in_channels, out_channels, scale_factor=2):
"""Transposed convolution (learnable upsampling)"""
kernel_size = 2 * scale_factor
stride = scale_factor
conv = torch.nn.ConvTranspose2d(in_channels, out_channels, kernel_size, stride)
return conv(x)
# Test
x = torch.randn(2, 64, 16, 16)
x_up_bilinear = upsample_bilinear(x, scale_factor=2)
print(f"Original shape: {x.shape}, Upsampled: {x_up_bilinear.shape}")
assert x_up_bilinear.shape == (2, 64, 32, 32), "Should upsampled correctly"
print("✓ Upsampling working")
if __name__ == "__main__":
print("Lab 1: Upsampling - PASSED")### Lab 2: Skip Connection Concatenation
import torch
import torch.nn as nn
import torch.nn.functional as F
def skip_connection(encoder_feature, decoder_feature):
"""Concatenate encoder and decoder features"""
# Ensure same spatial resolution
if encoder_feature.shape[-2:] != decoder_feature.shape[-2:]:
decoder_feature = F.interpolate(decoder_feature, size=encoder_feature.shape[-2:], mode='bilinear', align_corners=False)
concatenated = torch.cat([encoder_feature, decoder_feature], dim=1)
return concatenated
# Test
encoder_feat = torch.randn(2, 256, 32, 32)
decoder_feat = torch.randn(2, 128, 32, 32)
skip_out = skip_connection(encoder_feat, decoder_feat)
print(f"Skip output shape: {skip_out.shape}")
assert skip_out.shape == (2, 384, 32, 32), "Should concatenate on channel dim"
print("✓ Skip connection working")
if __name__ == "__main__":
print("Lab 2: Skip Connection - PASSED")### Lab 3: Per-Pixel Cross-Entropy Loss
import torch
import torch.nn as nn
import numpy as np
def compute_segmentation_loss(logits, targets, class_weights=None):
"""Per-pixel cross-entropy loss"""
# logits: [B, C, H, W]
# targets: [B, H, W] (class indices)
criterion = nn.CrossEntropyLoss(weight=class_weights)
loss = criterion(logits, targets)
return loss
# Test
batch_size, n_classes, height, width = 4, 5, 32, 32
logits = torch.randn(batch_size, n_classes, height, width)
targets = torch.randint(0, n_classes, (batch_size, height, width))
loss = compute_segmentation_loss(logits, targets)
print(f"Segmentation loss: {loss:.4f}")
assert loss > 0, "Loss should be positive"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Segmentation loss working")
if __name__ == "__main__":
print("Lab 3: Segmentation Loss - PASSED")### Lab 4: Segmentation Metrics
import torch
import numpy as np
def compute_iou_per_class(predictions, targets, n_classes):
"""Compute IoU for each class; average for mIoU"""
ious = []
for c in range(n_classes):
pred_mask = (predictions == c)
target_mask = (targets == c)
intersection = (pred_mask & target_mask).sum()
union = (pred_mask | target_mask).sum()
iou = intersection / (union + 1e-8)
ious.append(iou.item())
return np.array(ious)
# Test
predictions = torch.randint(0, 5, (4, 32, 32))
targets = torch.randint(0, 5, (4, 32, 32))
ious = compute_iou_per_class(predictions, targets, n_classes=5)
print(f"IoU per class: {ious}")
print(f"mIoU: {ious.mean():.4f}")
assert len(ious) == 5, "Should have 5 IoUs"
assert all(0 <= iou <= 1 for iou in ious), "IoU should be in [0,1]"
print("✓ Segmentation metrics working")
if __name__ == "__main__":
print("Lab 4: Metrics - PASSED")