Image Segmentation Fully Convolutional Networks

# Image Segmentation & Fully Convolutional Networks

## Introduction & Motivation

Image Segmentation: assign class labels to pixels. Semantic and instance segmentation. Applications: medical imaging, autonomous driving, scene understanding.

Motivation: Pixel-level understanding; dense prediction.

Applications: Medical imaging, road scene understanding.

---

## Core Concepts & Theory

### Semantic Segmentation

Classify every pixel to single class.

### Instance Segmentation

Distinguish individual object instances.

### Fully Convolutional Networks (FCN)

End-to-end, pixels-to-pixels learning.

### U-Net

Encoder-decoder with skip connections.

---

## Mathematical Formulation

FCN Loss:
$$L = -\sum_{c=1}^{C} \sum_{i,j} w_{ij}^c \log(\hat{y}_{ij}^c)$$

Skip Connection:
$$f_ ext{combined} = f_ ext{decoder} + f_ ext{skip}$$

Dice Loss:
$$L = 1 - \frac{2|X \cap Y|}{|X| + |Y|}$$

---

## Advanced Theory & Extensions

### DeepLab

Atrous convolution; ASPP module.

### Mask R-CNN

Object detection + segmentation masks.

### PSPNet

Pyramid pooling for multi-scale context.

---

## Computational Considerations

FCN: O(H·W·C²).

Upsampling: Bilinear, transposed convolution.

Memory: O(H·W·C) for feature maps.

---

## Practical Implementation Strategies

### Multi-Scale Input

Process at different resolutions.

### CRF Post-processing

Refine segmentation boundaries.

### Class Weighting

Balance rare classes.

---

## Benchmark Datasets & Evaluation

Pascal VOC: 20 classes, segmentation masks.

Cityscapes: Autonomous driving, 19 classes.

ADE20K: 150 semantic classes.

---

## Key Challenges & Limitations

### Boundary Accuracy

Precise object boundaries.

### Class Imbalance

Rare classes underrepresented.

### Memory Requirements

Large feature maps for dense prediction.

---

## Hyperparameter Tuning

Atrous rate: 6, 12, 18.

Class weights: Inverse frequency weighting.

Dice weight: 0.3-0.5.

---

## Real-World Applications & Case Studies

Medical Imaging: Tumor segmentation in CT/MRI.

Autonomous Driving: Road, sidewalk, vehicle segmentation.

Satellite Imagery: Land cover classification.

---

## Integration with Other Methods

Segmentation + instance detection for full scene understanding; + edge detection for boundary refinement.

---

## Summary & Key Takeaways

Image Segmentation via FCN and encoder-decoder architectures enables dense pixel-level prediction.

Principles:
1. FCN: End-to-end learning.
2. Skip connections: Feature fusion.
3. Upsampling: Resolution restoration.
4. Multi-scale: Contextual information.
5. Loss weighting: Class balance.

---

---

## Appendix: Practical Labs

### Lab 1: Dice Loss

import numpy as np

def dice_loss(predictions, targets, smooth=1e-6):
 """Compute Dice loss for segmentation"""
 predictions = (predictions > 0.5).astype(float)
 targets = targets.astype(float)
 
 intersection = np.sum(predictions * targets)
 union = np.sum(predictions) + np.sum(targets)
 
 dice = (2.0 * intersection + smooth) / (union + smooth)
 loss = 1.0 - dice
 
 return loss

# Test
np.random.seed(42)
preds = np.random.rand(64, 256, 256)
targets = np.random.randint(0, 2, (64, 256, 256))

loss = dice_loss(preds, targets)

assert 0 <= loss <= 1, "Loss in range"
print("✓ Dice loss working")

if __name__ == "__main__":
 print("Lab 1: DiceLoss - PASSED")

### Lab 2: Skip Connection Fusion

import numpy as np

def fuse_skip_connections(encoder_features, decoder_features, method='add'):
 """Fuse encoder skip connections with decoder"""
 if method == 'add':
 fused = encoder_features + decoder_features
 elif method == 'concat':
 fused = np.concatenate([encoder_features, decoder_features], axis=-1)
 elif method == 'multiply':
 fused = encoder_features * decoder_features
 
 return fused

# Test
np.random.seed(42)
encoder = np.random.randn(2, 64, 64, 256)
decoder = np.random.randn(2, 64, 64, 256)

fused_add = fuse_skip_connections(encoder, decoder, 'add')
fused_concat = fuse_skip_connections(encoder, decoder, 'concat')

assert fused_add.shape == encoder.shape, "Add shape"
assert fused_concat.shape[:-1] == encoder.shape[:-1], "Concat shape"
print("✓ Skip connection fusion working")

if __name__ == "__main__":
 print("Lab 2: SkipConnectionFusion - PASSED")

### Lab 3: IoU Metric

import numpy as np

def compute_iou_mask(predictions, targets, num_classes):
 """Compute Intersection over Union for segmentation"""
 ious = []
 
 predictions = np.argmax(predictions, axis=-1)
 targets = np.argmax(targets, axis=-1)
 
 for c in range(num_classes):
 intersection = np.sum((predictions == c) & (targets == c))
 union = np.sum((predictions == c) | (targets == c))
 
 iou = intersection / (union + 1e-8)
 ious.append(iou)
 
 mean_iou = np.mean(ious)
 return mean_iou

# Test
np.random.seed(42)
preds = np.random.randn(2, 256, 256, 21)
targets = np.random.randn(2, 256, 256, 21)

miou = compute_iou_mask(preds, targets, 21)

assert 0 <= miou <= 1, "mIoU in range"
print("✓ IoU metric working")

if __name__ == "__main__":
 print("Lab 3: IoUMetric - PASSED")

### Lab 4: Upsampling Operations

import numpy as np

def upsample_bilinear(feature_map, scale_factor=2):
 """Bilinear upsampling"""
 h, w = feature_map.shape
 new_h, new_w = h * scale_factor, w * scale_factor
 
 upsampled = np.zeros((new_h, new_w, feature_map.shape[-1]))
 
 for i in range(new_h):
 for j in range(new_w):
 src_i = i / scale_factor
 src_j = j / scale_factor
 
 # Bilinear interpolation (simplified)
 upsampled[i, j] = feature_map[int(src_i), int(src_j)]
 
 return upsampled

# Test
np.random.seed(42)
feature = np.random.randn(32, 32, 256)

upsampled = upsample_bilinear(feature, 2)

assert upsampled.shape == (64, 64, 256), "Upsampled shape"
print("✓ Upsampling working")

if __name__ == "__main__":
 print("Lab 4: Upsampling - PASSED")