Scene Decomposition is the task of breaking down a visual scene into its constituent components—individual objects, background, and their spatial relationships—enabling separate processing, reasoning, and manipulation of each element. In neural approaches, scene decomposition produces per-object masks, feature representations, and spatial parameters from a single input image or video, either through supervised segmentation or unsupervised object-centric learning.

Why Scene Decomposition Matters in AI/ML: Scene decomposition is a foundational capability for visual understanding that enables compositional reasoning, physics prediction, and scene editing by providing structured representations of scene content rather than entangled, holistic feature maps.

• Supervised decomposition — Instance segmentation (Mask R-CNN, SAM) and panoptic segmentation provide per-object masks using labeled training data; these methods achieve high accuracy on standard benchmarks but require expensive per-pixel annotation • Unsupervised decomposition — Object-centric methods (Slot Attention, MONet, IODINE) learn to decompose scenes using only reconstruction objectives; each component is represented by a separate latent vector and decoded independently • 3D-aware decomposition — Neural radiance fields (NeRF) variants decompose scenes into individual 3D objects with separate NeRFs per object, enabling novel view synthesis with per-object control (moving, removing, or editing individual objects) • Video decomposition — Temporal consistency across video frames provides strong cues for decomposition: objects maintain identity across frames, enabling tracking-based decomposition (SAVi, SIMONe) where motion patterns separate objects from background • Hierarchical decomposition — Scenes can be decomposed at multiple levels: scene → objects → parts → materials; hierarchical decomposition captures the recursive compositional structure of visual scenes

Scene decomposition is the essential perceptual capability that bridges low-level vision and high-level reasoning, transforming raw pixel inputs into structured, object-level representations that enable compositional understanding, per-object manipulation, and physics-based reasoning about the visual world.