Instance Segmentation Mask R-CNN
# Instance Segmentation: Mask R-CNN
## Introduction & Motivation
Instance segmentation: object detection + pixel-level masks. Mask R-CNN: extends Faster R-CNN with mask head. Region-of-Interest Align: precise spatial alignment. Applications: object detection, instance segmentation, keypoint detection.
Motivation: Semantic segmentation lacks instance information. Instance segmentation combines detection and segmentation.
Applications: Medical imaging, autonomous driving, robotics.
---
## Core Concepts & Theory
### Region of Interest Pooling
Extract features for each region.
### ROI Align
Bilinear interpolation; precise alignment.
### Mask Head
FCN on RoI features; pixel-level prediction.
---
## Mathematical Formulation
ROI Align:
$$ ext{ROIAlign}(X, ext{roi}) = ext{bilinear}( ext{sample}(X, ext{roi}))$$
Mask loss (binary cross-entropy per pixel):
$$L_{ ext{mask}} = -\frac{1}{K^2} \sum_i [m_i \log(\hat{m}_i) + (1-m_i) \log(1-\hat{m}_i)]$$
Combined loss:
$$L = L_{ ext{bbox}} + L_{ ext{cls}} + L_{ ext{mask}}$$
---
## Advanced Theory & Extensions
### Keypoint Detection
Keypoint heatmaps per ROI; human pose.
### Panoptic Segmentation
Combine instance + semantic.
### 3D Object Detection
Extend 2D boxes to 3D.
---
## Computational Considerations
Mask R-CNN: O(features + ROIAlign + mask_head) overhead.
ROI Align: Bilinear interpolation; O(N × roi_area).
Inference: ~200-400ms per image; GPU required.
---
## Practical Implementation Strategies
### ROI Sampling
Balance positive/negative ROIs; foreground focus.
### Multi-Scale Backbones
FPN for multi-scale detection.
### Data Augmentation
Scale, flip, rotate; segmentation-aware.
---
## Benchmark Datasets & Evaluation
COCO: Instance segmentation standard; mAP metrics.
Cityscapes: Autonomous driving; panoptic metrics.
Medical: Instance segmentation for organs/cells.
---
## Key Challenges & Limitations
### Computational Cost
Slower than detection-only; GPU memory intensive.
### Overlapping Masks
Confusion with overlapping instances.
### Small Instances
Difficult to segment accurately.
---
## Hyperparameter Tuning
Mask head depth: 4 layers typical; 256 channels.
ROI size: 14×14 standard; larger for small objects.
Mask pooling: ROIAlign > ROIPool.
---
## Real-World Applications & Case Studies
Medical Imaging: Organ instance segmentation.
Robotics: Object recognition and grasp planning.
Autonomous Driving: Instance-level scene understanding.
---
## Integration with Other Methods
Instance Seg + Tracking → multi-object tracking.
Instance Seg + 3D → 3D scene understanding.
---
## Summary & Key Takeaways
Instance segmentation via Mask R-CNN extends object detection with mask prediction through ROI Align and mask heads.
Principles:
1. Faster R-CNN: detection base.
2. ROI Align: spatial precision.
3. Mask head: pixel-level prediction.
4. Multi-scale: FPN features.
5. Joint training: shared backbone.
---
---
## Appendix: Practical Labs
### Lab 1: ROI Pooling
import numpy as np
def roi_pool(features, rois, output_size=7):
"""ROI pooling: extract features per ROI"""
B, C, H, W = features.shape
num_rois = len(rois)
pooled = np.zeros((num_rois, C, output_size, output_size))
for i, roi in enumerate(rois):
x1, y1, x2, y2 = roi
x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
# Clip to feature map
x1 = max(0, min(x1, W-1))
y1 = max(0, min(y1, H-1))
x2 = max(x1+1, min(x2, W))
y2 = max(y1+1, min(y2, H))
roi_features = features[:, :, y1:y2, x1:x2]
# Max pool to output size
if roi_features.shape[-1] > 0:
pooled[i] = np.mean(roi_features, axis=(0, 2, 3), keepdims=True)
return pooled
# Test
np.random.seed(42)
features = np.random.randn(1, 256, 14, 14)
rois = [[2, 2, 8, 8], [5, 5, 12, 12]]
pooled = roi_pool(features, rois)
assert pooled.shape == (2, 256, 7, 7), "Pooled shape"
print("✓ ROI pooling working")
if __name__ == "__main__":
print("Lab 1: ROIPooling - PASSED")### Lab 2: Mask Head
import numpy as np
class MaskHead:
def __init__(self, input_channels=256, num_classes=80):
self.input_channels = input_channels
self.num_classes = num_classes
def forward(self, roi_features):
"""Generate masks from ROI features"""
B, C, H, W = roi_features.shape
# Simplified mask prediction
masks = np.random.randn(B, self.num_classes, H, H)
return masks
# Test
np.random.seed(42)
mask_head = MaskHead(input_channels=256, num_classes=80)
roi_features = np.random.randn(32, 256, 7, 7)
masks = mask_head.forward(roi_features)
assert masks.shape[0] == 32, "Batch size"
assert masks.shape[1] == 80, "Num classes"
print("✓ Mask head working")
if __name__ == "__main__":
print("Lab 2: MaskHead - PASSED")### Lab 3: Mask IoU
import numpy as np
def compute_mask_iou(pred_mask, true_mask):
"""Compute IoU for binary masks"""
intersection = np.logical_and(pred_mask > 0.5, true_mask > 0.5).sum()
union = np.logical_or(pred_mask > 0.5, true_mask > 0.5).sum()
iou = intersection / (union + 1e-8)
return iou
# Test
np.random.seed(42)
pred = np.random.rand(28, 28)
true = np.random.rand(28, 28)
iou = compute_mask_iou(pred, true)
assert 0 <= iou <= 1, "Mask IoU in [0,1]"
print("✓ Mask IoU working")
if __name__ == "__main__":
print("Lab 3: MaskIoU - PASSED")### Lab 4: Instance Segmentation Metrics
import numpy as np
def compute_panoptic_quality(pred_masks, true_masks):
"""Simplified panoptic quality"""
# Match predicted to ground truth
max_iou = 0
matched = []
for pred in pred_masks:
best_iou = 0
best_gt = -1
for gt_idx, true in enumerate(true_masks):
if gt_idx in matched:
continue
intersection = np.logical_and(pred, true).sum()
union = np.logical_or(pred, true).sum()
iou = intersection / (union + 1e-8)
if iou > best_iou:
best_iou = iou
best_gt = gt_idx
if best_iou > 0.5 and best_gt >= 0:
matched.append(best_gt)
max_iou += best_iou
pq = max_iou / (len(pred_masks) + 1e-8)
return pq
# Test
np.random.seed(42)
pred_masks = [np.random.rand(28, 28) for _ in range(5)]
true_masks = [np.random.rand(28, 28) for _ in range(5)]
pq = compute_panoptic_quality(pred_masks, true_masks)
assert 0 <= pq <= 1, "PQ in [0,1]"
print("✓ Panoptic quality working")
if __name__ == "__main__":
print("Lab 4: PanopticQuality - PASSED")