Instance Segmentation Mask R-CNN Object-Level Masks

# Instance Segmentation: Mask R-CNN & Object-Level Masks

## Introduction & Motivation

Instance segmentation: per-object pixel masks. Mask R-CNN: extend Faster R-CNN with mask prediction branch. Separate object instances; high-quality boundaries. Applications: scene understanding, interactive segmentation, panoptic tasks.

Motivation: Semantic segmentation lacks instance identity. Instance segmentation provides per-object masks; enables object-level operations.

Applications: Scene parsing, interactive segmentation, video object tracking, robotic manipulation.

---

## Core Concepts & Theory

### Region-Based Approach

RPN proposes regions; classify + bbox + mask per region.

### Mask Prediction

Small FCN predicts binary mask per region; handles multiple objects.

### RoI Align

Precise spatial alignment; improves mask quality vs RoI Pooling.

---

## Mathematical Formulation

Mask R-CNN loss:
$$\mathcal{L} = \mathcal{L}_{ ext{cls}} + \mathcal{L}_{ ext{bbox}} + \mathcal{L}_{ ext{mask}}$$

$$\mathcal{L}_{ ext{mask}} = -\frac{1}{K m^2} \sum_{i,j} \left[ y_{ij} \log(\hat{y}_{ij}) + (1 - y_{ij}) \log(1 - \hat{y}_{ij}) ight]$$

RoI Align:
$$ ext{RoI}( ext{feature}, ext{bbox}) = ext{bilinear\_sample}( ext{feature}, ext{grid\_points})$$

---

## Advanced Theory & Extensions

### Keypoint Detection

Add keypoint branch; multi-task learning.

### Panoptic Segmentation

Combine instance + semantic via Mask R-CNN.

### Cascade R-CNN

Multi-stage refinement; progressive mask refinement.

---

## Computational Considerations

RPN: O(N proposals) classification + bbox.

Mask head: O(N proposals × m²) where m = mask resolution.

RoI Align: O(N proposals × grid_points) bilinear sampling.

---

## Practical Implementation Strategies

### Mask Resolution

Typical 14×14 or 28×28; balance accuracy-speed.

### Multi-Scale Heads

Apply mask head at multiple feature levels.

### Post-Processing

Morphological ops; boundary refinement.

---

## Benchmark Datasets & Evaluation

COCO: Instance segmentation; 80 classes, 2.5M instances.

Cityscapes: Driving; instance + semantic combined.

Metrics: AP (Average Precision), AP50, AP75.

---

## Key Challenges & Limitations

### Boundary Quality

Masks may leak at instance boundaries; refinement needed.

### Computational Cost

3-4× slower than bounding box detection.

### Small Objects

Difficult to segment small instances; scale sensitivity.

---

## Hyperparameter Tuning

Mask resolution: 14-28; higher = better but slower.

RoI scale: 224×224 for region crops.

Mask loss weight: 1.0; equal to bbox.

---

## Real-World Applications & Case Studies

Instance Segmentation: COCO benchmark standard solution.

Interactive Segmentation: User clicks guide mask refinement.

Robotic Manipulation: Object masks for grasping points.

---

## Integration with Other Methods

Mask R-CNN + Tracking → video instance tracking.

Mask R-CNN + 3D → 3D instance understanding.

---

## Summary & Key Takeaways

Instance segmentation via Mask R-CNN extends object detection with pixel-level masks, enabling fine-grained object-level understanding.

Principles:
1. Region-based: proposals → classify, bbox, mask.
2. RoI Align: precise spatial alignment via bilinear sampling.
3. Multi-task loss: classification + bbox + mask branches.
4. Mask head: FCN predicts binary masks per region.
5. Post-processing: morphological ops refine boundaries.

---

---

## Appendix: Practical Labs

### Lab 1: Mask Prediction Head

import torch
import torch.nn as nn

class MaskHead(nn.Module):
 def __init__(self, in_channels=256, hidden_dim=256, mask_size=14):
 super().__init__()
 self.conv1 = nn.Conv2d(in_channels, hidden_dim, 3, padding=1)
 self.conv2 = nn.Conv2d(hidden_dim, hidden_dim, 3, padding=1)
 self.deconv = nn.ConvTranspose2d(hidden_dim, hidden_dim, 2, stride=2)
 self.mask_conv = nn.Conv2d(hidden_dim, 1, 1)
 
 def forward(self, x):
 x = torch.relu(self.conv1(x))
 x = torch.relu(self.conv2(x))
 x = torch.relu(self.deconv(x))
 mask = torch.sigmoid(self.mask_conv(x))
 return mask

# Test
mask_head = MaskHead(in_channels=256, hidden_dim=256, mask_size=14)
features = torch.randn(8, 256, 14, 14)
masks = mask_head(features)

print(f"Mask output shape: {masks.shape}")
assert masks.shape == (8, 1, 28, 28), "Should output 28x28 masks"
assert (masks >= 0).all() and (masks <= 1).all(), "Mask should be in [0,1]"
print("✓ Mask head working")

if __name__ == "__main__":
 print("Lab 1: Mask Head - PASSED")

### Lab 2: RoI Align

import torch
import torch.nn.functional as F

def roi_align(features, rois, output_size=7):
 """RoI Align: bilinear sampling from features"""
 # features: [B, C, H, W]
 # rois: [N, 4] (x1, y1, x2, y2)
 # output_size: (h, w)
 
 b, c, h, w = features.shape
 n = len(rois)
 
 # Normalize roi coordinates to [-1, 1]
 rois_norm = rois.clone().float()
 rois_norm[:, [0, 2]] = (rois_norm[:, [0, 2]] / w) * 2 - 1
 rois_norm[:, [1, 3]] = (rois_norm[:, [1, 3]] / h) * 2 - 1
 
 # Create grid
 grid_h = torch.linspace(-1, 1, output_size, device=features.device)
 grid_w = torch.linspace(-1, 1, output_size, device=features.device)
 grid_y, grid_x = torch.meshgrid(grid_h, grid_w, indexing='ij')
 
 aligned = []
 for i in range(n):
 x1, y1, x2, y2 = rois_norm[i]
 # Scale grid to roi
 scaled_x = grid_x * (x2 - x1) / 2 + (x1 + x2) / 2
 scaled_y = grid_y * (y2 - y1) / 2 + (y1 + y2) / 2
 grid = torch.stack([scaled_x, scaled_y], dim=-1).unsqueeze(0)
 
 # Bilinear sampling
 roi_feat = F.grid_sample(features.unsqueeze(0), grid, align_corners=True)
 aligned.append(roi_feat.squeeze(0))
 
 return torch.cat(aligned, dim=0)

# Test
features = torch.randn(1, 256, 64, 64)
rois = torch.tensor([[10., 10., 30., 30.], [40., 40., 55., 55.]])

aligned_feats = roi_align(features, rois, output_size=7)

print(f"RoI aligned shape: {aligned_feats.shape}")
assert aligned_feats.shape == (2, 256, 7, 7), "Should output aligned features"
print("✓ RoI Align working")

if __name__ == "__main__":
 print("Lab 2: RoI Align - PASSED")

### Lab 3: Mask Loss

import torch
import torch.nn.functional as F
import numpy as np

def compute_mask_loss(pred_masks, gt_masks):
 """Binary cross-entropy loss for mask predictions"""
 # pred_masks: [N, 1, H, W] logits
 # gt_masks: [N, 1, H, W] binary targets
 
 loss = F.binary_cross_entropy_with_logits(pred_masks, gt_masks.float(), reduction='mean')
 return loss

# Test
pred = torch.randn(8, 1, 28, 28)
gt = torch.randint(0, 2, (8, 1, 28, 28))

loss = compute_mask_loss(pred, gt)

print(f"Mask loss: {loss:.4f}")
assert loss > 0, "Loss should be positive"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Mask loss working")

if __name__ == "__main__":
 print("Lab 3: Mask Loss - PASSED")

### Lab 4: Instance Metrics

import torch
import numpy as np

def compute_mask_iou(pred_mask, gt_mask):
 """Compute IoU for instance masks"""
 pred = (pred_mask > 0.5).float()
 gt = gt_mask.float()
 
 intersection = (pred * gt).sum()
 union = ((pred + gt) > 0).sum()
 
 iou = intersection / (union + 1e-8)
 return iou.item()

def evaluate_instance_masks(pred_masks, gt_masks):
 """Evaluate instance mask quality"""
 ious = []
 for pred, gt in zip(pred_masks, gt_masks):
 iou = compute_mask_iou(pred, gt)
 ious.append(iou)
 
 return np.array(ious)

# Test
pred = torch.sigmoid(torch.randn(10, 1, 28, 28))
gt = torch.randint(0, 2, (10, 1, 28, 28))

ious = evaluate_instance_masks(pred, gt)

print(f"Mean mask IoU: {ious.mean():.4f}")
assert len(ious) == 10, "Should have 10 IoUs"
assert all(0 <= iou <= 1 for iou in ious), "IoU should be in [0,1]"
print("✓ Instance metrics working")

if __name__ == "__main__":
 print("Lab 4: Metrics - PASSED")