Panoptic Segmentation - Unified Framework
# Panoptic Segmentation - Unified Framework
## Introduction & Motivation
Panoptic segmentation: unified semantic + instance segmentation. Predict class labels and instance IDs. Applications: scene understanding, autonomous driving.
Motivation: Unified framework for both semantic and instance-level understanding.
Applications: Autonomous driving, scene parsing, robotics.
---
## Core Concepts & Theory
### Semantic Segmentation Branch
Per-pixel class prediction.
### Instance Segmentation Branch
Per-object mask prediction.
### Panoptic Head
Combine semantic and instance outputs.
### Stuff and Things
Distinguish background classes from objects.
---
## Mathematical Formulation
Panoptic Loss:
$$\mathcal{L} = \mathcal{L}_{ ext{semantic}} + \lambda \mathcal{L}_{ ext{instance}}$$
Panoptic Quality:
$$ ext{PQ} = \frac{\sum_i ext{IoU}_i}{ ext{TP} + 0.5 ext{FP} + 0.5 ext{FN}}$$
Segmentation Quality:
$$ ext{SQ} = \frac{\sum_i ext{IoU}_i}{ ext{TP}}$$
---
## Advanced Theory & Extensions
### Post-Processing Rules
Merge semantic and instance.
### Multi-Scale Inference
Process at different resolutions.
### Boundary Refinement
Improve edge predictions.
---
## Computational Considerations
Semantic branch: O(H·W·C).
Instance branch: O(N·h·w·D).
Merging: O(H·W·(C+N)).
---
## Practical Implementation Strategies
### Feature Sharing
Share backbone between branches.
### Overlapping Handling
Resolve conflicts at boundaries.
### Threshold Tuning
Balance semantic and instance.
---
## Benchmark Datasets & Evaluation
COCO Panoptic: Full panoptic benchmark.
Cityscapes Panoptic: Urban scenes.
ADE20K Panoptic: Complex scenes.
---
## Key Challenges & Limitations
### Stuff vs Things
Clear distinction needed.
### Boundary Accuracy
Edge prediction difficult.
### Computational Cost
Dual branch overhead.
---
## Hyperparameter Tuning
Loss weight: 0.5-2.0.
Semantic threshold: 0.5-0.9.
Instance threshold: 0.5-0.7.
---
## Real-World Applications & Case Studies
Autonomous Driving: Road scene understanding.
Robotics: Scene perception for manipulation.
Video Analysis: Temporal consistency.
---
## Integration with Other Methods
Panoptic + video segmentation; + temporal consistency for video understanding.
---
## Summary & Key Takeaways
Panoptic segmentation unifies semantic and instance understanding.
Principles:
1. Dual branches: Semantic and instance.
2. Unified framework: Single model for both.
3. Stuff and things: Distinguish classes.
4. Panoptic quality: Evaluate both aspects.
5. Scalability: Multi-scale processing.
---
## Appendix: Practical Labs
### Lab 1: Merge Semantic and Instance
import numpy as np
def merge_panoptic(semantic_pred, instance_pred, num_classes):
"""Merge semantic and instance predictions"""
panoptic = np.zeros_like(semantic_pred) * 1000 + semantic_pred
# Merge instance IDs for instance classes
for class_id in range(num_classes):
mask = semantic_pred == class_id
instances = instance_pred * mask
# Encode instance ID in high bits
panoptic[mask] = (class_id * 1000) + instances[mask]
return panoptic
np.random.seed(42)
sem = np.random.randint(0, 10, (256, 256))
inst = np.random.randint(0, 100, (256, 256))
panop = merge_panoptic(sem, inst, 10)
assert panop.shape == sem.shape
print("✓ Merge panoptic working")### Lab 2: Stuff vs Things Classification
def classify_stuff_things(class_id, stuff_classes):
"""Classify if class is stuff or things"""
if class_id in stuff_classes:
return 'stuff'
else:
return 'things'
# Typical: sky, grass, wall are stuff; car, person are things
stuff = {1, 2, 3, 4, 5}
things = {6, 7, 8, 9}
assert classify_stuff_things(2, stuff) == 'stuff'
assert classify_stuff_things(6, stuff) == 'things'
print("✓ Stuff vs things classification working")### Lab 3: Panoptic Quality Computation
import numpy as np
def compute_panoptic_quality(pred_panoptic, gt_panoptic):
"""Compute PQ metric"""
pred_ids = set(np.unique(pred_panoptic)) - {0}
gt_ids = set(np.unique(gt_panoptic)) - {0}
tp, fp, fn = 0, 0, 0
iou_sum = 0
for pred_id in pred_ids:
pred_mask = pred_panoptic == pred_id
# Find best matching GT
best_iou = 0
best_gt_id = None
for gt_id in gt_ids:
gt_mask = gt_panoptic == gt_id
intersection = np.logical_and(pred_mask, gt_mask).sum()
union = np.logical_or(pred_mask, gt_mask).sum()
iou = intersection / (union + 1e-8)
if iou > best_iou:
best_iou = iou
best_gt_id = gt_id
if best_iou > 0.5:
tp += 1
iou_sum += best_iou
else:
fp += 1
fn = len(gt_ids) - tp
pq = iou_sum / max(tp + 0.5*fp + 0.5*fn, 1)
return pq
np.random.seed(42)
pred = np.random.randint(0, 100, (256, 256))
gt = np.random.randint(0, 100, (256, 256))
pq = compute_panoptic_quality(pred, gt)
assert 0 <= pq <= 1
print(f"✓ Panoptic quality: {pq:.3f}")### Lab 4: Boundary Extraction
import numpy as np
def extract_boundaries(panoptic_pred):
"""Extract object boundaries"""
boundaries = np.zeros_like(panoptic_pred)
# Compute edges via gradient
for i in range(1, panoptic_pred.shape[0]-1):
for j in range(1, panoptic_pred.shape[1]-1):
neighbors = [panoptic_pred[i-1,j], panoptic_pred[i+1,j],
panoptic_pred[i,j-1], panoptic_pred[i,j+1]]
if panoptic_pred[i,j] not in neighbors:
boundaries[i, j] = 1
return boundaries
np.random.seed(42)
panop = np.random.randint(0, 20, (128, 128))
bounds = extract_boundaries(panop)
assert bounds.sum() > 0
print(f"✓ Boundaries extracted: {bounds.sum()} pixels")---