relighting

**Relighting** is the process of **changing the lighting in images or 3D scenes** — modifying illumination conditions to simulate different times of day, weather, or artificial lighting, enabling realistic lighting edits for photography, film, AR, and virtual production without recapturing the scene. **What Is Relighting?** - **Definition**: Modify lighting in captured images or scenes. - **Input**: Image/scene + desired lighting conditions. - **Output**: Image/scene with new lighting. - **Goal**: Realistic lighting changes without physical recapture. **Why Relighting?** - **Photography**: Change lighting after capture (golden hour, studio lighting). - **Film/VFX**: Match lighting across shots, create dramatic effects. - **AR/VR**: Realistic virtual objects matching real lighting. - **Virtual Production**: Real-time lighting changes on LED stages. - **E-Commerce**: Show products under different lighting conditions. **Relighting Approaches** **Image-Based Relighting**: - **Method**: Modify image appearance to simulate new lighting. - **Techniques**: Intrinsic decomposition, neural relighting. - **Benefit**: Works with single image. - **Limitation**: Limited to plausible lighting changes. **Geometry-Based Relighting**: - **Method**: Reconstruct 3D geometry, relight using rendering. - **Pipeline**: 3D reconstruction → material estimation → rendering with new lights. - **Benefit**: Physically accurate, flexible lighting. - **Challenge**: Requires accurate geometry and materials. **Light Stage Capture**: - **Method**: Capture subject under many lighting conditions. - **Relight**: Linearly combine captured images for any lighting. - **Benefit**: Photorealistic, accurate. - **Challenge**: Requires expensive light stage equipment. **Neural Relighting**: - **Method**: Neural networks learn to relight images. - **Training**: Learn from multi-illumination datasets. - **Benefit**: Fast, works with single image. - **Examples**: Neural Relighting, Deep Relighting Networks. **Relighting Techniques** **Intrinsic Image Decomposition**: - **Method**: Separate reflectance and shading. - **Relight**: Modify shading component, keep reflectance. - **Benefit**: Lighting-independent material editing. **Spherical Harmonics**: - **Method**: Represent lighting as spherical harmonic coefficients. - **Relight**: Change coefficients to modify lighting. - **Benefit**: Compact representation, efficient. **Environment Map Relighting**: - **Method**: Use environment maps (HDR images) for lighting. - **Relight**: Replace environment map. - **Benefit**: Realistic global illumination. **Neural Rendering**: - **Method**: Neural networks render scene under new lighting. - **Training**: Learn light transport from data. - **Benefit**: Fast, handles complex effects. **Applications** **Portrait Photography**: - **Use**: Change lighting on portraits after capture. - **Examples**: Studio lighting, golden hour, dramatic lighting. - **Benefit**: Flexibility without reshoots. **Product Photography**: - **Use**: Show products under different lighting. - **Benefit**: Consistent lighting across product catalog. **Film and VFX**: - **Use**: Match lighting across shots, create effects. - **Examples**: Day-for-night, time of day changes. - **Benefit**: Creative control in post-production. **Augmented Reality**: - **Use**: Match virtual object lighting to real scene. - **Benefit**: Realistic AR integration. **Virtual Production**: - **Use**: Real-time relighting on LED stages. - **Benefit**: In-camera final pixels, reduced post-production. **Relighting Challenges** **Shadows**: - **Problem**: Changing lighting requires changing shadows. - **Challenge**: Realistic shadow synthesis. - **Solution**: Geometry-aware methods, learned shadow generation. **Specularities**: - **Problem**: Highlights change with lighting direction. - **Challenge**: View-dependent effects. - **Solution**: BRDF estimation, physics-based rendering. **Inter-Reflections**: - **Problem**: Light bounces between surfaces. - **Challenge**: Global illumination effects. - **Solution**: Path tracing, neural rendering. **Occlusions**: - **Problem**: New lighting may reveal occluded regions. - **Challenge**: Inpainting hidden areas. - **Solution**: Multi-view capture, learned priors. **Relighting Pipeline** **Image-Based**: 1. **Intrinsic Decomposition**: Separate reflectance and shading. 2. **Lighting Estimation**: Estimate current lighting. 3. **Shading Synthesis**: Generate new shading for target lighting. 4. **Recomposition**: Combine reflectance with new shading. **Geometry-Based**: 1. **3D Reconstruction**: Recover scene geometry. 2. **Material Estimation**: Estimate surface materials (BRDF). 3. **Lighting Specification**: Define new lighting (environment map, point lights). 4. **Rendering**: Render scene with new lighting. **Neural**: 1. **Input**: Image + target lighting parameters. 2. **Network**: Neural network predicts relit image. 3. **Output**: Relit image. **Relighting Methods** **One Light At a Time (OLAT)**: - **Capture**: Photograph subject with one light at a time. - **Relight**: Linearly combine images for any lighting. - **Benefit**: Accurate, flexible. - **Challenge**: Requires many captures (100+). **Polynomial Texture Maps (PTM)**: - **Method**: Fit polynomial to pixel intensity vs. light direction. - **Relight**: Evaluate polynomial for new light direction. - **Benefit**: Compact, efficient. **Reflectance Transfer**: - **Method**: Transfer lighting from one image to another. - **Use**: Match lighting across images. **Deep Learning Relighting**: - **Method**: Train neural networks on multi-illumination data. - **Examples**: Deep Relighting Networks, Neural Relighting. - **Benefit**: Single image input, fast inference. **Quality Metrics** - **PSNR**: Peak signal-to-noise ratio. - **SSIM**: Structural similarity. - **LPIPS**: Learned perceptual similarity. - **User Studies**: Subjective realism assessment. - **Shadow Accuracy**: Correctness of shadow placement and softness. **Relighting Datasets** **Multi-Illumination**: - **MIT Intrinsic Images**: Objects under multiple lighting. - **Light Stage Data**: Faces captured in light stages. **Synthetic**: - **Rendered Scenes**: 3D scenes rendered with different lighting. - **Benefit**: Perfect ground truth. **Relighting Tools** **Commercial**: - **Adobe Photoshop**: Basic relighting tools. - **Substance Painter**: Material-based relighting. - **Unreal Engine**: Real-time relighting for virtual production. **Research**: - **Neural Relighting**: Deep learning-based methods. - **Light Stage**: Professional capture systems. **Future of Relighting** - **Single-Image**: Accurate relighting from single image. - **Real-Time**: Interactive relighting for live applications. - **Video**: Temporally consistent relighting for video. - **Semantic**: Understand scene semantics for better relighting. - **Generalization**: Models that work on any scene. Relighting is **essential for modern visual content creation** — it enables flexible lighting control after capture, supporting applications from photography to film to augmented reality, making lighting a creative tool rather than a constraint.