Neural Radiance Fields (NeRF) and 3D Gaussian Splatting is a class of neural 3D scene representation methods that synthesize photorealistic novel views of scenes from a sparse set of input photographs — revolutionizing 3D reconstruction and rendering by replacing traditional mesh-based or point-cloud pipelines with learned volumetric or primitive-based representations.

NeRF: Neural Radiance Fields

NeRF (Mildenhall et al., 2020) represents a 3D scene as a continuous volumetric function mapping 5D input (3D position x,y,z + 2D viewing direction θ,φ) to color (RGB) and density (σ) using a multilayer perceptron (MLP). Rendering proceeds via volume rendering: rays are cast from camera pixels through the scene, sampled at discrete points along each ray, and accumulated using alpha compositing. The MLP is trained by minimizing photometric loss between rendered and ground-truth images. Positional encoding (Fourier features) maps low-dimensional inputs to high-dimensional space, enabling the MLP to represent high-frequency detail.

NeRF Training and Rendering Pipeline

• Input: 20-100 posed photographs with known camera intrinsics and extrinsics (estimated via COLMAP structure-from-motion)
• Ray marching: 64-256 sample points per ray; hierarchical sampling (coarse + fine networks) concentrates samples near surfaces
• Training time: Original NeRF requires 1-2 days per scene on a single GPU; optimized via Instant-NGP (NVIDIA) to minutes using hash grid encoding
• Rendering speed: Original NeRF renders at ~0.05 FPS (minutes per frame); Instant-NGP achieves interactive rates (~15 FPS)
• Mip-NeRF: Anti-aliased NeRF using integrated positional encoding over conical frustums rather than point samples, improving multi-scale rendering quality

NeRF Extensions and Variants

• Dynamic NeRF: D-NeRF, Nerfies, and HyperNeRF extend to deformable and dynamic scenes by conditioning on time or learned deformation fields
• Generative NeRF: DreamFusion (Google) and Magic3D (NVIDIA) generate 3D objects from text prompts via score distillation sampling from 2D diffusion models
• Large-scale NeRF: Block-NeRF and Mega-NeRF scale to city-level scenes by partitioning space into blocks with separate NeRFs
• Few-shot NeRF: PixelNeRF and MVSNeRF generalize across scenes from 1-3 input views using learned priors from multi-view datasets
• Surface extraction: NeuS and VolSDF extract explicit mesh surfaces from NeRF representations using signed distance functions (SDF)

3D Gaussian Splatting

• Explicit representation: Represents scenes as millions of 3D Gaussian primitives, each defined by position (mean), covariance (shape/orientation), opacity, and spherical harmonic coefficients (view-dependent color)
• Rasterization-based rendering: Projects Gaussians onto the image plane and alpha-blends in depth order—no ray marching required
• Training: Starts from COLMAP sparse point cloud; Gaussians are optimized via gradient descent on photometric loss; adaptive density control splits large Gaussians and removes transparent ones
• Real-time rendering: Achieves 100+ FPS at 1080p resolution using custom CUDA rasterizer—orders of magnitude faster than NeRF
• Quality: Matches or exceeds NeRF quality on standard benchmarks (Mip-NeRF 360, Tanks and Temples) while training in 10-30 minutes

3D Gaussian Splatting Advances

• Dynamic Gaussians: 4D Gaussian Splatting adds temporal deformation for dynamic scene reconstruction from monocular video
• Compression: Compact-3DGS and other methods reduce storage from hundreds of MB to tens of MB via quantization and pruning of Gaussian parameters
• SLAM integration: Gaussian splatting as the scene representation for real-time simultaneous localization and mapping (MonoGS, SplaTAM)
• Avatar generation: Animatable Gaussians for real-time human avatar rendering from monocular video
• Text-to-3D: GaussianDreamer and DreamGaussian generate 3D Gaussian scenes from text or image prompts in minutes

Applications and Industry Impact

• Virtual reality and telepresence: Real-time novel view synthesis enables immersive VR experiences from captured scenes
• Digital twins: High-fidelity 3D reconstructions of buildings, factories, and infrastructure for monitoring and simulation
• E-commerce: Product visualization from a small number of photographs with realistic relighting
• Film and gaming: Asset creation from real-world captures, reducing manual 3D modeling effort

Neural 3D representations have transformed computer vision and graphics, with 3D Gaussian Splatting's real-time rendering capability making photorealistic novel view synthesis practical for interactive applications that were previously impossible with traditional or NeRF-based approaches.