Semantic Segmentation Architectures

# Semantic Segmentation Architectures

## Introduction & Motivation

Semantic segmentation: pixel-level classification. FCN, U-Net, DeepLab. Applications: scene understanding, medical imaging.

Motivation: Classify every pixel in an image.

Applications: Autonomous driving, medical image analysis.

---

## Core Concepts & Theory

### Fully Convolutional Networks

End-to-end convolutional layers.

### Encoder-Decoder Architecture

Downsampling then upsampling.

### Skip Connections

Feature map concatenation.

### Dilated Convolution

Expand receptive field without pooling.

---

## Mathematical Formulation

FCN Output:
$$ ext{out} = ext{upsample}( ext{conv}(x))$$

U-Net Skip:
$$x_{decode} = ext{concat}(x_{encode}, x_{up})$$

Dilated Convolution:
$$y[i,j] = \sum_m \sum_n w[m,n] \cdot x[i + m \cdot d, j + n \cdot d]$$

---

## Advanced Theory & Extensions

### Atrous Spatial Pyramid Pooling

Multi-scale dilated convolutions.

### DeepLab Architecture

Atrous convolution and CRF refinement.

### PSPNet

Pyramid pooling module.

---

## Computational Considerations

Encoder: O(H·W·C·K²).

Decoder upsampling: O(H·W·C).

Skip concatenation: O(H·W·C).

---

## Practical Implementation Strategies

### Output Stride

Control resolution via downsampling factor.

### CRF Post-Processing

Refine boundaries with conditional random fields.

### Multi-Scale Testing

Inference at multiple scales.

---

## Benchmark Datasets & Evaluation

PASCAL VOC: Semantic segmentation standard.

Cityscapes: Autonomous driving scenes.

ADE20K: Scene parsing benchmark.

---

## Key Challenges & Limitations

### Boundary Accuracy

Exact edge delineation.

### Class Imbalance

Rare class learning.

### Computational Cost

High-resolution processing.

---

## Hyperparameter Tuning

Atrous rates: 6, 12, 18.

Output stride: 8 or 16.

Learning rate: 1e-4 to 1e-2.

---

## Real-World Applications & Case Studies

Medical Segmentation: Organ and tumor identification.

Scene Parsing: Autonomous driving understanding.

Satellite Imagery: Land cover classification.

---

## Integration with Other Methods

Segmentation + CRF for refinement; + attention for focus.

---

## Summary & Key Takeaways

Semantic segmentation assigns class labels to image pixels.

Principles:
1. FCN: Fully convolutional approach.
2. Encoder-decoder: Spatial hierarchy preservation.
3. Skip connections: Feature reuse.
4. Dilated convolution: Receptive field expansion.
5. Multi-scale: Hierarchical processing.

---

## Appendix: Practical Labs

### Lab 1: Upsampling via Bilinear Interpolation

import numpy as np

def bilinear_upsample(x, scale_factor=2):
 """Bilinear interpolation for upsampling"""
 h, w = x.shape
 h_new = h * scale_factor
 w_new = w * scale_factor
 
 upsampled = np.zeros((h_new, w_new))
 
 for i in range(h_new):
 for j in range(w_new):
 src_i = i / scale_factor
 src_j = j / scale_factor
 
 i0, i1 = int(src_i), min(int(src_i) + 1, h - 1)
 j0, j1 = int(src_j), min(int(src_j) + 1, w - 1)
 
 di = src_i - i0
 dj = src_j - j0
 
 upsampled[i, j] = (1 - di) * (1 - dj) * x[i0, j0]
 upsampled[i, j] += di * (1 - dj) * x[i1, j0]
 upsampled[i, j] += (1 - di) * dj * x[i0, j1]
 upsampled[i, j] += di * dj * x[i1, j1]
 
 return upsampled

np.random.seed(42)
x = np.random.randn(7, 7)
upsampled = bilinear_upsample(x, scale_factor=2)
assert upsampled.shape[0] > x.shape[0], "Upsampled correctly"
print("✓ Bilinear upsampling working")

### Lab 2: Skip Connection Concatenation

import numpy as np

def skip_connection(encoder_features, decoder_features):
 """Concatenate encoder and decoder features"""
 if encoder_features.shape != decoder_features.shape:
 raise ValueError("Shape mismatch")
 
 concatenated = np.concatenate([encoder_features, decoder_features], axis=2)
 return concatenated

np.random.seed(42)
enc_feat = np.random.randn(64, 64, 128)
dec_feat = np.random.randn(64, 64, 128)
skip = skip_connection(enc_feat, dec_feat)
assert skip.shape[2] == 256, "Correct concatenation"
print("✓ Skip connection working")

### Lab 3: Dilated Convolution

import numpy as np

def dilated_conv2d(x, w, dilation=2, stride=1):
 """2D dilated convolution"""
 h, w_size = w.shape
 h_img, w_img = x.shape[:2]
 
 h_out = (h_img - (h - 1) * dilation - 1) // stride + 1
 w_out = (w_img - (w_size - 1) * dilation - 1) // stride + 1
 
 out = np.zeros((h_out, w_out))
 
 for i in range(h_out):
 for j in range(w_out):
 h_start = i * stride
 w_start = j * stride
 patch = np.zeros((h, w_size))
 for di in range(h):
 for dj in range(w_size):
 patch[di, dj] = x[h_start + di * dilation, w_start + dj * dilation]
 out[i, j] = np.sum(patch * w)
 
 return out

np.random.seed(42)
x = np.random.randn(32, 32)
w = np.random.randn(3, 3)
out = dilated_conv2d(x, w, dilation=2)
assert out.shape[0] > 0, "Dilated convolution working"
print("✓ Dilated convolution working")

### Lab 4: Pixel-wise Softmax

import numpy as np

def segmentation_softmax(logits, num_classes=21):
 """Softmax over spatial dimensions"""
 h, w = logits.shape[:2]
 
 logits = logits - np.max(logits, axis=2, keepdims=True)
 exp_logits = np.exp(logits)
 probs = exp_logits / np.sum(exp_logits, axis=2, keepdims=True)
 
 return probs

np.random.seed(42)
logits = np.random.randn(64, 64, 21)
probs = segmentation_softmax(logits)
assert probs.shape == logits.shape, "Correct probability shape"
assert np.allclose(probs.sum(axis=2), 1.0), "Valid probabilities"
print("✓ Segmentation softmax working")

---