3d gaussian splatting

**3D Gaussian Splatting** is a **novel 3D scene representation using anisotropic 3D Gaussians** — representing scenes as collections of oriented ellipsoids that can be rendered extremely fast through rasterization, achieving real-time rendering speeds (100+ FPS) while maintaining quality comparable to NeRF, revolutionizing real-time photorealistic rendering. **What Is 3D Gaussian Splatting?** - **Definition**: Represent 3D scenes as sets of 3D Gaussian primitives. - **Primitive**: Each Gaussian is an oriented ellipsoid with position, covariance, color, opacity. - **Rendering**: Fast rasterization-based rendering (not ray marching). - **Speed**: 100-200 FPS real-time rendering on consumer GPUs. - **Quality**: Comparable to NeRF, often better for fine details. **Why Gaussian Splatting?** **Speed**: - **Real-Time**: 100+ FPS rendering (vs. 1-30 FPS for NeRF variants). - **Rasterization**: Leverages GPU rasterization pipeline. - **No Ray Marching**: Avoids expensive volumetric integration. **Quality**: - **High Fidelity**: Photorealistic rendering quality. - **Fine Details**: Captures thin structures better than NeRF. - **View-Dependent**: Supports view-dependent effects. **Flexibility**: - **Explicit**: Gaussians can be edited, moved, deleted. - **Interpretable**: Each Gaussian has clear geometric meaning. **3D Gaussian Representation** **Gaussian Primitive**: - **Position**: μ = (x, y, z) — center of Gaussian. - **Covariance**: Σ — 3x3 matrix defining shape and orientation. - **Color**: c = (r, g, b) or spherical harmonics for view-dependence. - **Opacity**: α — transparency. **Gaussian Function**: ``` G(x) = exp(-1/2 (x - μ)^T Σ^-1 (x - μ)) Where: - x: 3D point - μ: Gaussian center - Σ: Covariance matrix (defines ellipsoid shape) ``` **Anisotropic**: - Gaussians are ellipsoids, not spheres. - Can be stretched and oriented to match scene geometry. - More efficient representation than isotropic Gaussians. **How Gaussian Splatting Works** **Training**: 1. **Initialization**: Start with sparse point cloud (from SfM). 2. **Optimization**: Optimize Gaussian parameters to match training images. - Position, covariance, color, opacity. 3. **Adaptive Density Control**: Add/remove Gaussians as needed. - Split large Gaussians in high-detail areas. - Remove low-opacity Gaussians. 4. **Convergence**: Train for 7k-30k iterations (minutes). **Rendering**: 1. **Projection**: Project 3D Gaussians to 2D screen space. 2. **Sorting**: Sort Gaussians by depth (front to back). 3. **Rasterization**: Rasterize each Gaussian as 2D splat. 4. **Alpha Blending**: Blend Gaussians using alpha compositing. 5. **Output**: Final rendered image. **Rendering Equation**: ``` C = Σ c_i α_i Π (1 - α_j) i j