Instance Segmentation Mask R-CNN Object-Level Masks
# Instance Segmentation: Mask R-CNN & Object-Level Masks
## Introduction & Motivation
Instance segmentation: per-object pixel masks. Mask R-CNN: extend Faster R-CNN with mask prediction branch. Separate object instances; high-quality boundaries. Applications: scene understanding, interactive segmentation, panoptic tasks.
Motivation: Semantic segmentation lacks instance identity. Instance segmentation provides per-object masks; enables object-level operations.
Applications: Scene parsing, interactive segmentation, video object tracking, robotic manipulation.
---
## Core Concepts & Theory
### Region-Based Approach
RPN proposes regions; classify + bbox + mask per region.
### Mask Prediction
Small FCN predicts binary mask per region; handles multiple objects.
### RoI Align
Precise spatial alignment; improves mask quality vs RoI Pooling.
---
## Mathematical Formulation
Mask R-CNN loss:
$$\mathcal{L} = \mathcal{L}_{ ext{cls}} + \mathcal{L}_{ ext{bbox}} + \mathcal{L}_{ ext{mask}}$$
$$\mathcal{L}_{ ext{mask}} = -\frac{1}{K m^2} \sum_{i,j} \left[ y_{ij} \log(\hat{y}_{ij}) + (1 - y_{ij}) \log(1 - \hat{y}_{ij}) ight]$$
RoI Align:
$$ ext{RoI}( ext{feature}, ext{bbox}) = ext{bilinear\_sample}( ext{feature}, ext{grid\_points})$$
---
## Advanced Theory & Extensions
### Keypoint Detection
Add keypoint branch; multi-task learning.
### Panoptic Segmentation
Combine instance + semantic via Mask R-CNN.
### Cascade R-CNN
Multi-stage refinement; progressive mask refinement.
---
## Computational Considerations
RPN: O(N proposals) classification + bbox.
Mask head: O(N proposals × m²) where m = mask resolution.
RoI Align: O(N proposals × grid_points) bilinear sampling.
---
## Practical Implementation Strategies
### Mask Resolution
Typical 14×14 or 28×28; balance accuracy-speed.
### Multi-Scale Heads
Apply mask head at multiple feature levels.
### Post-Processing
Morphological ops; boundary refinement.
---
## Benchmark Datasets & Evaluation
COCO: Instance segmentation; 80 classes, 2.5M instances.
Cityscapes: Driving; instance + semantic combined.
Metrics: AP (Average Precision), AP50, AP75.
---
## Key Challenges & Limitations
### Boundary Quality
Masks may leak at instance boundaries; refinement needed.
### Computational Cost
3-4× slower than bounding box detection.
### Small Objects
Difficult to segment small instances; scale sensitivity.
---
## Hyperparameter Tuning
Mask resolution: 14-28; higher = better but slower.
RoI scale: 224×224 for region crops.
Mask loss weight: 1.0; equal to bbox.
---
## Real-World Applications & Case Studies
Instance Segmentation: COCO benchmark standard solution.
Interactive Segmentation: User clicks guide mask refinement.
Robotic Manipulation: Object masks for grasping points.
---
## Integration with Other Methods
Mask R-CNN + Tracking → video instance tracking.
Mask R-CNN + 3D → 3D instance understanding.
---
## Summary & Key Takeaways
Instance segmentation via Mask R-CNN extends object detection with pixel-level masks, enabling fine-grained object-level understanding.
Principles:
1. Region-based: proposals → classify, bbox, mask.
2. RoI Align: precise spatial alignment via bilinear sampling.
3. Multi-task loss: classification + bbox + mask branches.
4. Mask head: FCN predicts binary masks per region.
5. Post-processing: morphological ops refine boundaries.
---
---
## Appendix: Practical Labs
### Lab 1: Mask Prediction Head
import torch
import torch.nn as nn
class MaskHead(nn.Module):
def __init__(self, in_channels=256, hidden_dim=256, mask_size=14):
super().__init__()
self.conv1 = nn.Conv2d(in_channels, hidden_dim, 3, padding=1)
self.conv2 = nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1)
self.deconv = nn.ConvTranspose2d(hidden_dim, hidden_dim, 2, stride=2)
self.mask_conv = nn.Conv2d(hidden_dim, 1, 1)
def forward(self, x):
x = torch.relu(self.conv1(x))
x = torch.relu(self.conv2(x))
x = torch.relu(self.deconv(x))
mask = torch.sigmoid(self.mask_conv(x))
return mask
# Test
mask_head = MaskHead(in_channels=256, hidden_dim=256, mask_size=14)
features = torch.randn(8, 256, 14, 14)
masks = mask_head(features)
print(f"Mask output shape: {masks.shape}")
assert masks.shape == (8, 1, 28, 28), "Should output 28x28 masks"
assert (masks >= 0).all() and (masks <= 1).all(), "Mask should be in [0,1]"
print("✓ Mask head working")
if __name__ == "__main__":
print("Lab 1: Mask Head - PASSED")### Lab 2: RoI Align
import torch
import torch.nn.functional as F
def roi_align(features, rois, output_size=7):
"""RoI Align: bilinear sampling from features"""
# features: [B, C, H, W]
# rois: [N, 4] (x1, y1, x2, y2)
# output_size: (h, w)
b, c, h, w = features.shape
n = len(rois)
# Normalize roi coordinates to [-1, 1]
rois_norm = rois.clone().float()
rois_norm[:, [0, 2]] = (rois_norm[:, [0, 2]] / w) * 2 - 1
rois_norm[:, [1, 3]] = (rois_norm[:, [1, 3]] / h) * 2 - 1
# Create grid
grid_h = torch.linspace(-1, 1, output_size, device=features.device)
grid_w = torch.linspace(-1, 1, output_size, device=features.device)
grid_y, grid_x = torch.meshgrid(grid_h, grid_w, indexing='ij')
aligned = []
for i in range(n):
x1, y1, x2, y2 = rois_norm[i]
# Scale grid to roi
scaled_x = grid_x * (x2 - x1) / 2 + (x1 + x2) / 2
scaled_y = grid_y * (y2 - y1) / 2 + (y1 + y2) / 2
grid = torch.stack([scaled_x, scaled_y], dim=-1).unsqueeze(0)
# Bilinear sampling
roi_feat = F.grid_sample(features.unsqueeze(0), grid, align_corners=True)
aligned.append(roi_feat.squeeze(0))
return torch.cat(aligned, dim=0)
# Test
features = torch.randn(1, 256, 64, 64)
rois = torch.tensor([[10., 10., 30., 30.], [40., 40., 55., 55.]])
aligned_feats = roi_align(features, rois, output_size=7)
print(f"RoI aligned shape: {aligned_feats.shape}")
assert aligned_feats.shape == (2, 256, 7, 7), "Should output aligned features"
print("✓ RoI Align working")
if __name__ == "__main__":
print("Lab 2: RoI Align - PASSED")### Lab 3: Mask Loss
import torch
import torch.nn.functional as F
import numpy as np
def compute_mask_loss(pred_masks, gt_masks):
"""Binary cross-entropy loss for mask predictions"""
# pred_masks: [N, 1, H, W] logits
# gt_masks: [N, 1, H, W] binary targets
loss = F.binary_cross_entropy_with_logits(pred_masks, gt_masks.float(), reduction='mean')
return loss
# Test
pred = torch.randn(8, 1, 28, 28)
gt = torch.randint(0, 2, (8, 1, 28, 28))
loss = compute_mask_loss(pred, gt)
print(f"Mask loss: {loss:.4f}")
assert loss > 0, "Loss should be positive"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Mask loss working")
if __name__ == "__main__":
print("Lab 3: Mask Loss - PASSED")### Lab 4: Instance Metrics
import torch
import numpy as np
def compute_mask_iou(pred_mask, gt_mask):
"""Compute IoU for instance masks"""
pred = (pred_mask > 0.5).float()
gt = gt_mask.float()
intersection = (pred * gt).sum()
union = ((pred + gt) > 0).sum()
iou = intersection / (union + 1e-8)
return iou.item()
def evaluate_instance_masks(pred_masks, gt_masks):
"""Evaluate instance mask quality"""
ious = []
for pred, gt in zip(pred_masks, gt_masks):
iou = compute_mask_iou(pred, gt)
ious.append(iou)
return np.array(ious)
# Test
pred = torch.sigmoid(torch.randn(10, 1, 28, 28))
gt = torch.randint(0, 2, (10, 1, 28, 28))
ious = evaluate_instance_masks(pred, gt)
print(f"Mean mask IoU: {ious.mean():.4f}")
assert len(ious) == 10, "Should have 10 IoUs"
assert all(0 <= iou <= 1 for iou in ious), "IoU should be in [0,1]"
print("✓ Instance metrics working")
if __name__ == "__main__":
print("Lab 4: Metrics - PASSED")