Intrinsic image decomposition is the task of separating an image into intrinsic components — decomposing appearance into reflectance (albedo) and shading (illumination), enabling material editing, relighting, and understanding of scene properties independent of lighting conditions.
What Is Intrinsic Image Decomposition?
- Definition: Decompose image into reflectance and shading.
- Input: Single RGB image.
- Output:
- Reflectance (Albedo): Surface color/texture independent of lighting.
- Shading (Illumination): Lighting effects (shadows, highlights).
- Relationship: Image = Reflectance × Shading (in linear space).
Why Intrinsic Decomposition?
- Material Editing: Change surface colors without affecting lighting.
- Relighting: Change lighting while preserving materials.
- Object Recognition: Recognize objects independent of lighting.
- Augmented Reality: Realistic insertion of virtual objects.
- Computational Photography: Advanced photo editing.
Intrinsic Components
Reflectance (Albedo):
- Definition: Intrinsic surface color/texture.
- Properties: Independent of lighting, viewpoint.
- Example: Red ball has red reflectance regardless of lighting.
Shading (Illumination):
- Definition: Lighting effects on surface.
- Components: Direct illumination, shadows, inter-reflections.
- Properties: Depends on lighting, geometry, viewpoint.
Image Formation:
``
I(x) = R(x) · S(x)
Where:
- I(x): Observed image intensity at pixel x
- R(x): Reflectance (albedo)
- S(x): Shading (illumination)
``
Intrinsic Decomposition Approaches
Optimization-Based:
- Method: Formulate as energy minimization.
- Energy: Data term + priors (smoothness, sparsity).
- Priors:
- Reflectance is piecewise constant.
- Shading is smooth.
- Reflectance changes at texture edges, shading at geometry edges.
- Examples: Retinex, Intrinsic Images in the Wild.
Learning-Based:
- Method: Neural networks learn decomposition.
- Training: Supervised on synthetic or real data with ground truth.
- Examples: CGIntrinsics, IIW, ShapeNet Intrinsics.
- Benefit: Handle complex real-world images.
Physics-Based:
- Method: Model light transport, inverse rendering.
- Benefit: Physically accurate decomposition.
- Challenge: Requires scene geometry, material properties.
Challenges
Ill-Posed Problem:
- Ambiguity: Infinite (reflectance, shading) pairs can produce same image.
- Example: Dark reflectance + bright shading = bright reflectance + dark shading.
- Solution: Priors, constraints, learning from data.
Texture vs. Shading:
- Problem: Distinguish texture (reflectance) from shading.
- Example: Polka dots (texture) vs. shadows (shading).
- Solution: Multi-scale analysis, learned features.
Complex Lighting:
- Problem: Inter-reflections, subsurface scattering, transparency.
- Challenge: Simple reflectance × shading model insufficient.
Ground Truth:
- Problem: Difficult to obtain ground truth for real images.
- Solution: Synthetic data, multi-illumination capture, crowdsourcing.
Intrinsic Decomposition Methods
Retinex:
- Classic: Separate reflectance and illumination based on gradients.
- Assumption: Reflectance has sharp edges, illumination is smooth.
- Limitation: Oversimplified, doesn't handle complex scenes.
Intrinsic Images in the Wild (IIW):
- Method: Learn from sparse human annotations.
- Annotations: Relative reflectance judgments (same/different material).
- Benefit: Scalable annotation, real-world data.
CGIntrinsics:
- Training: Synthetic data from 3D scenes.
- Network: CNN predicts reflectance and shading.
- Benefit: Large-scale training data.
ShapeNet Intrinsics:
- Training: Rendered 3D objects with known reflectance/shading.
- Benefit: Perfect ground truth for training.
Applications
Material Editing:
- Use: Change surface colors independently of lighting.
- Example: Recolor walls, furniture, clothing.
- Benefit: Realistic edits respecting lighting.
Relighting:
- Use: Change lighting while preserving materials.
- Process: Decompose → modify shading → recompose.
- Example: Change time of day, add/remove lights.
Object Recognition:
- Use: Recognize objects from reflectance (lighting-invariant).
- Benefit: Robust to lighting variations.
Augmented Reality:
- Use: Understand scene lighting for realistic AR.
- Benefit: Virtual objects match real lighting.
Computational Photography:
- Use: Advanced photo editing (selective relighting, material transfer).
- Benefit: Physically plausible edits.
Intrinsic Decomposition Techniques
Multi-Illumination:
- Method: Capture scene under multiple lighting conditions.
- Benefit: Resolve ambiguities, accurate decomposition.
- Challenge: Requires controlled capture.
Multi-View:
- Method: Use multiple viewpoints.
- Benefit: Geometric constraints aid decomposition.
Video:
- Method: Temporal consistency across frames.
- Benefit: More constraints, better decomposition.
Semantic Guidance:
- Method: Use semantic segmentation to guide decomposition.
- Benefit: Material boundaries align with semantic boundaries.
Quality Metrics
MSE (Mean Squared Error):
- Definition: Pixel-wise error in reflectance and shading.
- Limitation: Doesn't account for perceptual quality.
LMSE (Local MSE):
- Definition: MSE after local scaling (handles scale ambiguity).
- Benefit: More robust to global intensity shifts.
DSSIM (Structural Dissimilarity):
- Definition: 1 - SSIM (structural similarity).
- Benefit: Perceptually motivated.
Intrinsic Decomposition Datasets
MIT Intrinsic Images:
- Data: Real objects with ground truth from multi-illumination capture.
- Size: Small but high-quality.
IIW (Intrinsic Images in the Wild):
- Data: Real images with sparse human annotations.
- Size: Large-scale, diverse scenes.
ShapeNet Intrinsics:
- Data: Rendered 3D objects with perfect ground truth.
- Size: Large-scale synthetic data.
MPI Sintel:
- Data: Animated movie frames with ground truth.
- Use: Evaluation on complex scenes.
Future of Intrinsic Decomposition
- Single-Image: Accurate decomposition from single image.
- Real-Time: Fast decomposition for interactive applications.
- Video: Temporally consistent decomposition.
- Semantic: Integrate semantic understanding.
- Physics-Based: Incorporate physical light transport models.
- Generalization: Models that work across diverse scenes.
Intrinsic image decomposition is fundamental to computational photography and computer vision — it enables understanding and manipulating images at the level of materials and lighting, supporting applications from photo editing to augmented reality to object recognition.