Semantic Segmentation Fcn Unet Deeplabv3
# Semantic Segmentation: FCN, UNet & DeepLabV3
## Introduction & Motivation
Semantic segmentation: pixel-level classification. FCN: fully convolutional; end-to-end learning. UNet: encoder-decoder with skip connections. DeepLabV3: atrous convolution; multi-scale context. Applications: medical imaging, autonomous driving, scene understanding.
Motivation: Image-level tasks insufficient; pixel-level understanding needed. Fully convolutional enables arbitrary input sizes.
Applications: Medical imaging, autonomous driving, scene parsing.
---
## Core Concepts & Theory
### Fully Convolutional Networks
Upsampling via deconvolution; learnable upsampling.
### Encoder-Decoder
Compress features; restore spatial resolution.
### Atrous (Dilated) Convolution
Increase receptive field; preserve resolution.
---
## Mathematical Formulation
Deconvolution upsampling:
$$y = ext{deconv}(x, ext{kernel}) + ext{bias}$$
Skip connection:
$$ ext{decoder}_l = ext{upsample}( ext{decoder}_{l+1}) + ext{encoder}_l$$
Atrous convolution:
$$y[i, j] = \sum_{k_1, k_2} w[k_1, k_2] \cdot x[i + d \cdot k_1, j + d \cdot k_2]$$
where d = dilation rate.
---
## Advanced Theory & Extensions
### Pyramid Pooling Module
Multi-scale pooling; PSPNET.
### Conditional Random Fields
Post-processing; spatial smoothness.
### Panoptic Segmentation
Instance + semantic; unified framework.
---
## Computational Considerations
FCN: O(H·W·C·D) per scale; O(H·W) outputs.
UNet: O(skip_connections) memory for residuals.
DeepLabV3: O(atrous_rate) varying receptive fields.
---
## Practical Implementation Strategies
### Multi-Scale Testing
Test at multiple scales; ensemble predictions.
### CRF Post-Processing
Enforce spatial smoothness; IoU improvement.
### Boundary Handling
Careful upsampling; minimize artifacts.
---
## Benchmark Datasets & Evaluation
Cityscapes: Autonomous driving standard; 19 classes.
PASCAL VOC: Classical benchmark; mIoU metric.
Medical Segmentation: Organ-specific datasets; Dice coefficient.
---
## Key Challenges & Limitations
### Class Imbalance
Rare classes; loss weighting or sampling.
### Boundary Precision
Difficult at object boundaries; edge-aware losses.
### Computational Efficiency
Pixel-level predictions expensive; real-time inference hard.
---
## Hyperparameter Tuning
Dilation rates: 6, 12, 18 typical; receptive field.
Encoder depth: ResNet-50, ResNet-101 common.
Output stride: 8, 16; resolution vs. computation tradeoff.
---
## Real-World Applications & Case Studies
Medical Imaging: Organ segmentation; FCN standard.
Autonomous Driving: Scene understanding; DeepLabV3 deployed.
Satellite Imagery: Land cover classification; semantic maps.
---
## Integration with Other Methods
Segmentation + Instance Seg → Panoptic.
Segmentation + Depth → 3D understanding.
---
## Summary & Key Takeaways
Semantic segmentation via FCN, UNet, and DeepLabV3 enables pixel-level understanding through fully convolutional, encoder-decoder, and atrous convolution architectures.
Principles:
1. FCN: fully convolutional; arbitrary sizes.
2. UNet: skip connections; symmetry.
3. DeepLabV3: atrous; multi-scale context.
4. CRF: spatial post-processing.
5. Evaluation: mIoU standard metric.
---
---
## Appendix: Practical Labs
### Lab 1: Deconvolution Upsampling
import numpy as np
def deconvolutional_upsample(x, kernel_size=4, stride=2):
"""Simplified deconvolution upsampling"""
B, C, H, W = x.shape
# Compute output size
H_out = (H - 1) * stride + kernel_size
W_out = (W - 1) * stride + kernel_size
# Initialize output
y = np.zeros((B, C, H_out, W_out))
# Deconvolution (simplified)
for b in range(B):
for c in range(C):
for i in range(H):
for j in range(W):
i_out = i * stride
j_out = j * stride
# Scatter operation
y[b, c, i_out:i_out+kernel_size, j_out:j_out+kernel_size] += x[b, c, i, j]
return y
# Test
np.random.seed(42)
x = np.random.randn(2, 64, 8, 8)
y = deconvolutional_upsample(x, kernel_size=4, stride=2)
assert y.shape[2] > x.shape[2], "Upsampled height"
assert y.shape[3] > x.shape[3], "Upsampled width"
print("✓ Deconvolutional upsampling working")
if __name__ == "__main__":
print("Lab 1: DeconvUpsample - PASSED")### Lab 2: Skip Connections
import numpy as np
def unet_decoder_with_skip(encoder_features, decoder_input):
"""UNet decoder with skip connections"""
# Simplified UNet decoder
x = decoder_input
# Upsample and add skip
x_up = 2 * x # Simplified upsampling
# Concatenate with encoder feature
skip_feature = encoder_features
# Crop/pad to match sizes
if x_up.shape[-1] != skip_feature.shape[-1]:
if x_up.shape[-1] > skip_feature.shape[-1]:
x_up = x_up[..., :skip_feature.shape[-1], :skip_feature.shape[-1]]
else:
pad = skip_feature.shape[-1] - x_up.shape[-1]
x_up = np.pad(x_up, ((0,0), (0,0), (0,pad), (0,pad)))
# Concatenate
x_fused = np.concatenate([x_up, skip_feature], axis=1)
return x_fused
# Test
np.random.seed(42)
encoder_feat = np.random.randn(2, 64, 16, 16)
decoder_in = np.random.randn(2, 128, 8, 8)
output = unet_decoder_with_skip(encoder_feat, decoder_in)
assert output.shape[1] == 64 + 128, "Concatenated channels"
print("✓ UNet skip connections working")
if __name__ == "__main__":
print("Lab 2: SkipConnections - PASSED")### Lab 3: Atrous Convolution
import numpy as np
def atrous_convolution(x, kernel, dilation_rate=1):
"""Atrous (dilated) convolution"""
B, C_in, H, W = x.shape
C_out, C_in, K, K = kernel.shape
# Dilate kernel
K_dilated = (K - 1) * dilation_rate + 1
output = np.zeros((B, C_out, H - K_dilated + 1, W - K_dilated + 1))
for b in range(B):
for c_out in range(C_out):
for i in range(output.shape[2]):
for j in range(output.shape[3]):
# Apply dilated kernel
receptive_field = 0
for ki in range(K):
for kj in range(K):
i_in = i + ki * dilation_rate
j_in = j + kj * dilation_rate
receptive_field += kernel[c_out, :, ki, kj] @ x[b, :, i_in, j_in]
output[b, c_out, i, j] = receptive_field
return output
# Test
np.random.seed(42)
x = np.random.randn(2, 3, 16, 16)
kernel = np.random.randn(64, 3, 3, 3)
y = atrous_convolution(x, kernel, dilation_rate=2)
assert y.shape[0] == 2, "Batch size"
assert y.shape[1] == 64, "Output channels"
print("✓ Atrous convolution working")
if __name__ == "__main__":
print("Lab 3: AtrousConv - PASSED")### Lab 4: Segmentation Metrics
import numpy as np
def compute_iou(pred_mask, true_mask):
"""Compute Intersection over Union"""
intersection = np.logical_and(pred_mask, true_mask).sum()
union = np.logical_or(pred_mask, true_mask).sum()
iou = intersection / (union + 1e-8)
return iou
def compute_dice(pred_mask, true_mask):
"""Compute Dice coefficient"""
intersection = np.logical_and(pred_mask, true_mask).sum()
dice = 2 * intersection / (pred_mask.sum() + true_mask.sum() + 1e-8)
return dice
# Test
np.random.seed(42)
pred = np.random.randint(0, 2, (256, 256))
true = np.random.randint(0, 2, (256, 256))
iou = compute_iou(pred, true)
dice = compute_dice(pred, true)
assert 0 <= iou <= 1, "IoU in [0,1]"
assert 0 <= dice <= 1, "Dice in [0,1]"
print("✓ Segmentation metrics working")
if __name__ == "__main__":
print("Lab 4: SegmentationMetrics - PASSED")