scene decomposition
**Scene Decomposition** is the task of breaking down a visual scene into its constituent components—individual objects, background, and their spatial relationships—enabling separate processing, reasoning, and manipulation of each element. In neural approaches, scene decomposition produces per-object masks, feature representations, and spatial parameters from a single input image or video, either through supervised segmentation or unsupervised object-centric learning.
**Why Scene Decomposition Matters in AI/ML:**
Scene decomposition is a **foundational capability for visual understanding** that enables compositional reasoning, physics prediction, and scene editing by providing structured representations of scene content rather than entangled, holistic feature maps.
• **Supervised decomposition** — Instance segmentation (Mask R-CNN, SAM) and panoptic segmentation provide per-object masks using labeled training data; these methods achieve high accuracy on standard benchmarks but require expensive per-pixel annotation
• **Unsupervised decomposition** — Object-centric methods (Slot Attention, MONet, IODINE) learn to decompose scenes using only reconstruction objectives; each component is represented by a separate latent vector and decoded independently
• **3D-aware decomposition** — Neural radiance fields (NeRF) variants decompose scenes into individual 3D objects with separate NeRFs per object, enabling novel view synthesis with per-object control (moving, removing, or editing individual objects)
• **Video decomposition** — Temporal consistency across video frames provides strong cues for decomposition: objects maintain identity across frames, enabling tracking-based decomposition (SAVi, SIMONe) where motion patterns separate objects from background
• **Hierarchical decomposition** — Scenes can be decomposed at multiple levels: scene → objects → parts → materials; hierarchical decomposition captures the recursive compositional structure of visual scenes
| Approach | Supervision | Output | Strengths |
|----------|------------|--------|-----------|
| Instance Segmentation | Per-pixel labels | Object masks + classes | High accuracy |
| Panoptic Segmentation | Per-pixel labels | Things + stuff masks | Complete coverage |
| Slot Attention | Reconstruction loss | Object slots + alpha masks | Unsupervised |
| SAM | Prompted/interactive | Instance masks | Zero-shot generalization |
| NeRF Decomposition | Multi-view images | 3D object representations | 3D-aware editing |
| Video Object Segmentation | First-frame mask | Tracked masks | Temporal consistency |
**Scene decomposition is the essential perceptual capability that bridges low-level vision and high-level reasoning, transforming raw pixel inputs into structured, object-level representations that enable compositional understanding, per-object manipulation, and physics-based reasoning about the visual world.**