Instance Segmentation - Mask R-CNN

# Instance Segmentation - Mask R-CNN

## Introduction & Motivation

Mask R-CNN: extend Faster R-CNN with instance segmentation. Per-object pixel masks. Applications: object detection with masks, instance-level understanding.

Motivation: Detect and segment individual object instances.

Applications: Scene understanding, robotic manipulation, medical image analysis.

---

## Core Concepts & Theory

### Region Proposal Network

Generate object proposals.

### RoI Align

Preserve spatial information in pooling.

### Mask Branch

Add segmentation head to Faster R-CNN.

### Multi-Task Learning

Joint detection and segmentation.

---

## Mathematical Formulation

RoI Align:
$$ ext{pool}(i,j) = \frac{1}{n^2} \sum_{(x,y) \in ext{bin}} f(x,y)$$

Mask Loss:
$$\mathcal{L}_{ ext{mask}} = -\sum_{i,j} \log \hat{m}_{ij}$$

Combined Loss:
$$\mathcal{L} = \mathcal{L}_{ ext{cls}} + \mathcal{L}_{ ext{box}} + \mathcal{L}_{ ext{mask}}$$

---

## Advanced Theory & Extensions

### Cascade R-CNN

Multi-stage refinement.

### FPN Integration

Feature pyramid for multi-scale.

### Panoptic Segmentation

Combine instance and semantic.

---

## Computational Considerations

RPN: O(H·W·A).

RoI pooling: O(N·R²·D²).

Mask branch: O(N·h·w·D).

---

## Practical Implementation Strategies

### Anchor Scales

Multi-scale region proposals.

### Non-Maximum Suppression

Remove overlapping boxes.

### Mask Threshold

Binarize mask predictions.

---

## Benchmark Datasets & Evaluation

COCO: Instance segmentation benchmark.

LVIS: Long-tail instance segmentation.

Cityscapes: Urban scene instances.

---

## Key Challenges & Limitations

### Small Objects

Detection and segmentation difficulty.

### Overlapping Instances

Occlusion handling.

### Computational Cost

Multi-stage inference expensive.

---

## Hyperparameter Tuning

RPN anchor scales: 32, 64, 128, 256, 512.

NMS threshold: 0.7.

Mask threshold: 0.5.

---

## Real-World Applications & Case Studies

Robotic Manipulation: Grasp point detection.

Medical Imaging: Tumor segmentation.

Video Object Tracking: Track instances across frames.

---

## Integration with Other Methods

Mask R-CNN + panoptic segmentation; + video consistency for temporal stability.

---

## Summary & Key Takeaways

Mask R-CNN extends Faster R-CNN with instance-level segmentation.

Principles:
1. Region proposals: RPN generates candidates.
2. RoI align: Preserve spatial precision.
3. Mask branch: Per-instance segmentation.
4. Multi-task: Joint detection-segmentation.
5. Scalability: FPN for multi-scale.

---

## Appendix: Practical Labs

### Lab 1: RoI Align

import numpy as np

def roi_align(feature_map, roi, pool_size=(7, 7)):
 """Align and pool region of interest"""
 x1, y1, x2, y2 = roi
 roi_height = y2 - y1
 roi_width = x2 - x1
 
 # Bilinear interpolation sampling
 output = np.zeros((pool_size[0], pool_size[1], feature_map.shape[2]))
 
 for i in range(pool_size[0]):
 for j in range(pool_size[1]):
 py = y1 + (i + 0.5) * roi_height / pool_size[0]
 px = x1 + (j + 0.5) * roi_width / pool_size[1]
 # Simplified: nearest neighbor
 output[i, j] = feature_map[int(py), int(px)]
 
 return output

np.random.seed(42)
feat = np.random.randn(224, 224, 256)
roi = [50, 50, 150, 150]
pooled = roi_align(feat, roi)
assert pooled.shape == (7, 7, 256)
print("✓ RoI Align working")

### Lab 2: Mask Loss

import numpy as np

def mask_loss(predicted_masks, ground_truth_masks):
 """Compute binary cross-entropy loss for masks"""
 loss = -np.mean(ground_truth_masks * np.log(predicted_masks + 1e-8) +
 (1 - ground_truth_masks) * np.log(1 - predicted_masks + 1e-8))
 return loss

np.random.seed(42)
pred = np.random.rand(10, 28, 28)
gt = np.random.rand(10, 28, 28) > 0.5
loss = mask_loss(pred, gt.astype(float))
assert loss > 0
print(f"✓ Mask loss: {loss:.3f}")

### Lab 3: NMS for Instances

import numpy as np

def nms_instances(boxes, scores, iou_threshold=0.5):
 """Non-maximum suppression for instance boxes"""
 indices = np.argsort(scores)[::-1]
 keep = []
 
 while len(indices) > 0:
 current = indices[0]
 keep.append(current)
 indices = indices[1:]
 
 if len(indices) == 0:
 break
 
 # Compute IoU with current box
 ious = []
 for i in indices:
 iou = compute_iou(boxes[current], boxes[i])
 ious.append(iou)
 
 indices = indices[np.array(ious) < iou_threshold]
 
 return keep

def compute_iou(box1, box2):
 intersection = max(0, min(box1[2], box2[2]) - max(box1[0], box2[0])) * \
 max(0, min(box1[3], box2[3]) - max(box1[1], box2[1]))
 union = (box1[2]-box1[0])*(box1[3]-box1[1]) + (box2[2]-box2[0])*(box2[3]-box2[1]) - intersection
 return intersection / union if union > 0 else 0

np.random.seed(42)
boxes = np.array([[10, 10, 50, 50], [12, 12, 52, 52], [100, 100, 150, 150]])
scores = np.array([0.9, 0.8, 0.7])
keep = nms_instances(boxes, scores)
assert len(keep) <= len(boxes)
print("✓ NMS working")

### Lab 4: Panoptic Quality Metric

import numpy as np

def panoptic_quality(pred_seg, gt_seg, num_classes):
 """Compute panoptic quality metric"""
 tp, fp, fn = 0, 0, 0
 iou_sum = 0
 
 for class_id in range(num_classes):
 pred_mask = (pred_seg == class_id)
 gt_mask = (gt_seg == class_id)
 
 intersection = np.logical_and(pred_mask, gt_mask).sum()
 union = np.logical_or(pred_mask, gt_mask).sum()
 
 if union > 0:
 iou = intersection / union
 iou_sum += iou
 if iou > 0.5:
 tp += 1
 else:
 fp += 1
 if intersection == 0 and union > 0:
 fn += 1
 
 pq = iou_sum / max(tp + 0.5*fp + 0.5*fn, 1)
 return pq

np.random.seed(42)
pred = np.random.randint(0, 10, (256, 256))
gt = np.random.randint(0, 10, (256, 256))
pq = panoptic_quality(pred, gt, 10)
assert 0 <= pq <= 1
print(f"✓ Panoptic Quality: {pq:.3f}")

---