Semantic Segmentation - Dense Prediction

# Semantic Segmentation - Dense Prediction

## Introduction & Motivation

Semantic segmentation: pixel-wise class prediction. Dense prediction for scene understanding. Applications: autonomous driving, medical imaging, scene parsing.

Motivation: Dense per-pixel classification for detailed scene understanding.

Applications: Autonomous driving, medical image analysis, scene parsing.

---

## Core Concepts & Theory

### Encoder-Decoder Architecture

Feature extraction and upsampling.

### Atrous Convolution

Dilated convolutions for receptive field.

### Skip Connections

Preserve spatial information.

### CRF Refinement

Post-processing with spatial consistency.

---

## Mathematical Formulation

Atrous Convolution:
$$y[i] = \sum_k x[i + r \cdot k] \cdot w[k]$$

Segmentation Loss:
$$\mathcal{L} = -\sum_i \sum_c y_{ic} \log(\hat{y}_{ic})$$

CRF Energy:
$$E(Y|X) = \sum_i \psi_u(y_i) + \sum_{ij} \psi_p(y_i, y_j)$$

---

## Advanced Theory & Extensions

### DeepLab Architecture

Atrous convolution and ASPP.

### Context Aggregation

Multi-scale feature fusion.

### Boundary Refinement

Improve segmentation edges.

---

## Computational Considerations

Forward pass: O(H·W·C·K²).

Memory: O(H·W·D).

CRF inference: O(H·W·C²).

---

## Practical Implementation Strategies

### Multi-Scale Input

Process at different resolutions.

### CRF Post-processing

Enforce spatial consistency.

### Data Augmentation

Random crops, flips, color jitter.

---

## Benchmark Datasets & Evaluation

Cityscapes: Urban driving scenes.

ADE20K: Scene parsing.

PASCAL VOC: General segmentation.

---

## Key Challenges & Limitations

### Long-Range Dependencies

Context limitations.

### Boundary Accuracy

Edge segmentation difficult.

### Computational Cost

Dense prediction expensive.

---

## Hyperparameter Tuning

Dilation rate: 1, 6, 12, 18.

Output stride: 8 or 16.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Autonomous Driving: Road scene understanding.

Medical Imaging: Lesion segmentation.

Satellite Imagery: Land-use classification.

---

## Integration with Other Methods

Segmentation + boundary detection; + instance segmentation for object-level precision.

---

## Summary & Key Takeaways

Semantic segmentation achieves dense scene understanding.

Principles:
1. Encoder-decoder: Feature extraction and upsampling.
2. Atrous convolution: Large receptive field.
3. Multi-scale: ASPP module.
4. Skip connections: Preserve details.
5. CRF refinement: Enforce consistency.

---

## Appendix: Practical Labs

### Lab 1: Atrous Convolution

import numpy as np

def atrous_convolution(input_map, kernel, dilation_rate=1):
 """Apply atrous (dilated) convolution"""
 h, w = input_map.shape[:2]
 kh, kw = kernel.shape
 
 output = np.zeros((h - (kh-1)*dilation_rate, w - (kw-1)*dilation_rate))
 
 for i in range(output.shape[0]):
 for j in range(output.shape[1]):
 receptive_field = input_map[i:i+(kh-1)*dilation_rate+1:dilation_rate,
 j:j+(kw-1)*dilation_rate+1:dilation_rate]
 output[i, j] = np.sum(receptive_field * kernel)
 
 return output

np.random.seed(42)
inp = np.random.randn(32, 32)
kern = np.random.randn(3, 3)
out = atrous_convolution(inp, kern, dilation_rate=2)
assert out.shape[0] > 0
print("✓ Atrous convolution working")

### Lab 2: Skip Connections

import numpy as np

def skip_connection(encoder_feat, decoder_feat):
 """Combine encoder and decoder features"""
 # Resize decoder to match encoder spatial dims if needed
 combined = encoder_feat + decoder_feat
 return combined

np.random.seed(42)
enc = np.random.randn(64, 64, 256)
dec = np.random.randn(64, 64, 256)
out = skip_connection(enc, dec)
assert out.shape == enc.shape
print("✓ Skip connection working")

### Lab 3: CRF Energy Minimization

import numpy as np

def crf_energy(predictions, labels, spatial_weight=1.0):
 """Compute CRF energy for spatial consistency"""
 # Unary potential
 unary = -np.log(predictions[np.arange(len(labels)), labels] + 1e-8)
 
 # Pairwise potential (simplified)
 pairwise = 0
 for i in range(len(labels)-1):
 if labels[i] != labels[i+1]:
 pairwise += spatial_weight
 
 energy = np.sum(unary) + pairwise
 return energy

np.random.seed(42)
preds = np.random.dirichlet(np.ones(10), size=20)
labels = np.random.randint(0, 10, 20)
energy = crf_energy(preds, labels)
assert energy >= 0
print(f"✓ CRF energy: {energy:.2f}")

### Lab 4: IoU Metric

import numpy as np

def iou_score(prediction, ground_truth):
 """Compute Intersection over Union"""
 intersection = np.logical_and(prediction, ground_truth).sum()
 union = np.logical_or(prediction, ground_truth).sum()
 iou = intersection / (union + 1e-8)
 return iou

np.random.seed(42)
pred = np.random.rand(256, 256) > 0.5
gt = np.random.rand(256, 256) > 0.5
iou = iou_score(pred, gt)
assert 0 <= iou <= 1
print(f"✓ IoU score: {iou:.3f}")

---