Neural Radiance Fields Novel View Synthesis

# Neural Radiance Fields & Novel View Synthesis

## Introduction & Motivation

Novel view synthesis is the task of generating a photorealistic image of a scene from a camera viewpoint that was never directly captured, given only a sparse set of observed photographs of that scene from other viewpoints. For decades this problem was approached through explicit geometric reconstruction: recovering a mesh, point cloud, or voxel grid representing the scene's 3D structure, then rendering that explicit representation from the desired new viewpoint. Neural Radiance Fields (NeRF), introduced by Mildenhall and colleagues in 2020, proposed a radically different approach: represent the entire scene implicitly as a continuous function, parameterized by a neural network, that maps any 3D spatial location and viewing direction to a color and a volume density, and render new views by numerically integrating this function along camera rays using classical volume rendering mathematics.

This implicit, coordinate-based representation proved remarkably effective at capturing fine geometric and appearance detail, including view-dependent effects such as specular highlights and reflections that explicit mesh-based representations struggle to reproduce naturally, and NeRF's striking visual results (particularly its ability to synthesize highly detailed, temporally consistent video-like camera flythroughs of a static scene from just a few dozen input photographs) triggered an explosion of follow-on research spanning faster training and rendering, dynamic and deformable scenes, generative and text-conditioned 3D content creation, and large-scale scene capture.

The practical significance of this line of work extends across visual effects and film production (virtual camera moves through real captured locations), e-commerce and virtual try-on (photorealistic 3D product visualization from photographs), robotics and autonomous driving (dense 3D scene representations useful for simulation and planning), cultural heritage preservation (digitizing physical sites and artifacts as explorable 3D scenes), and immersive telepresence and virtual reality content creation, all of which benefit from the ability to convert a modest set of ordinary photographs into a fully explorable, photorealistic 3D representation of a real scene.

More recently, 3D Gaussian Splatting has emerged as an influential alternative representation that retains much of NeRF's visual fidelity while enabling real-time rendering rates, representing scenes as a large collection of explicit, differentiable 3D Gaussian primitives rather than an implicit neural function, illustrating that the underlying goal (photorealistic, differentiable scene representation learned from posed photographs) can be pursued through substantially different representational choices, each with distinct trade-offs in rendering speed, training time, memory footprint, and editability.

## Core Concepts & Theory

A radiance field is a continuous function that, for every point in 3D space and every possible viewing direction from that point, specifies the emitted or reflected color of light in that direction, along with a volume density describing how much that point in space attenuates and scatters light passing through it (conceptually similar to the density of fog or smoke: higher density means light is more strongly blocked or scattered at that location). NeRF represents this radiance field with a multilayer perceptron that takes a 3D position and a 2D viewing direction as input and outputs an RGB color and a scalar density.

Because a standard multilayer perceptron struggles to represent the high-frequency spatial detail present in real scenes when given raw, low-dimensional coordinate inputs (a well-documented bias of coordinate-based networks toward learning low-frequency functions, sometimes called spectral bias), NeRF applies a positional encoding to the input coordinates before feeding them to the network, mapping each scalar coordinate to a vector of sine and cosine functions evaluated at exponentially increasing frequencies. This encoding dramatically improves the network's ability to represent fine spatial detail, and the choice of maximum encoding frequency directly controls the finest level of geometric and textural detail the resulting radiance field can represent.

Rendering an image from the learned radiance field proceeds by casting a ray from the camera through each pixel, sampling a set of points along that ray, querying the neural network at each sampled point to obtain its color and density, and then compositing these per-point colors and densities into a single pixel color using the classical volume rendering integral from computer graphics, which accumulates color along the ray while accounting for occlusion (points closer to the camera that have high density block the contribution of points further away). This entire pipeline, from ray sampling through network evaluation through volume compositing, is fully differentiable, which is what allows the network's parameters to be optimized directly by comparing rendered pixel colors against the observed pixel colors in the input photographs and backpropagating the resulting photometric error.

The training data for a NeRF model consists of a set of photographs of a static scene along with the precise camera pose (position and orientation) from which each photograph was captured, typically recovered using structure-from-motion software such as COLMAP as a preprocessing step, since the volume rendering optimization requires knowing exactly which 3D ray in space corresponds to each observed pixel.

## Mathematical Formulation

The core volume rendering equation used by NeRF computes the expected color of a camera ray r(t) = o + t*d (parameterized by a starting point o, a direction d, and a distance t along the ray) by integrating the color contributions of all points along the ray, weighted by how much of the light emitted at each point survives (is not absorbed or occluded) on its way back to the camera:

$$ C(r) = \int_{t_n}^{t_f} T(t) \, \sigma(r(t)) \, c(r(t), d) \, dt $$

where sigma of r(t) is the volume density at the 3D point r(t), c of r(t) and d is the emitted color at that point given the viewing direction d, and T(t) is the accumulated transmittance, representing the probability that a ray successfully travels from the near bound t_n to the point t without being absorbed by any intervening density:

$$ T(t) = \exp\left(-\int_{t_n}^{t} \sigma(r(s)) \, ds ight) $$

In practice, this continuous integral is approximated numerically using a discrete set of sampled points along the ray via a quadrature rule, converting the integral into a finite weighted sum over N sampled points with spacing delta_i between consecutive samples:

$$ \hat{C}(r) = \sum_{i=1}^{N} T_i \left(1 - \exp(-\sigma_i \delta_i) ight) c_i, \qquad T_i = \exp\left(-\sum_{j=1}^{i-1} \sigma_j \delta_j ight) $$

which is the discrete rendering formula actually implemented and differentiated through during NeRF training, with the training loss simply being the sum of squared differences between this rendered color and the true observed pixel color across all rays sampled from the training photographs, summed with an additional loss term from a coarser, hierarchically sampled network used to guide where the finer network concentrates its sampling along each ray (the original NeRF's coarse-to-fine hierarchical sampling scheme).

## Advanced Theory & Extensions

Positional encoding, applied separately to the input 3D position and viewing direction before either is passed to the network, maps each input scalar to a vector of sinusoids at a range of frequencies, a specific instance of what is now understood more generally, through the lens of neural tangent kernel theory, as controlling the effective smoothness or frequency content that a coordinate-based network can represent: without this encoding, standard multilayer perceptrons are strongly biased toward representing only smooth, low-frequency functions of their input coordinates, regardless of network depth or width, and this bias must be explicitly counteracted for the network to capture fine scene detail.

Instant Neural Graphics Primitives (Instant-NGP) dramatically accelerated NeRF training, from the many hours required by the original formulation down to seconds, by replacing the purely coordinate-based positional encoding with a learned multi-resolution hash-grid feature encoding, in which spatial coordinates are used to look up and interpolate learned feature vectors stored in a set of hash tables at multiple resolutions, moving much of the representational burden from the neural network's weights into a much faster, more directly optimizable spatial data structure, while retaining a small neural network to convert the looked-up features into the final color and density outputs.

3D Gaussian Splatting represents a scene not as an implicit neural function evaluated by ray marching, but as an explicit, initially sparse set of 3D Gaussian primitives (each with a learned position, covariance matrix controlling its size and orientation, opacity, and color, often represented via spherical harmonics to capture view-dependent appearance), which are rendered by projecting (splatting) each 3D Gaussian onto the 2D image plane and alpha-compositing the resulting 2D footprints in depth order, an approach that avoids the expensive per-pixel ray marching of implicit NeRF representations and enables real-time rendering rates on consumer graphics hardware while achieving comparable or superior visual fidelity, at the cost of a generally larger memory footprint (storing millions of explicit Gaussian primitives) compared to a compact implicit network.

Dynamic and deformable extensions of both NeRF and Gaussian Splatting add a time or deformation dimension to the representation, either by conditioning the radiance field on a time variable directly, or by learning a canonical (reference-pose) representation alongside a separate deformation field that warps sampled points from the observation frame back into the canonical frame before querying the radiance field, enabling reconstruction of moving or deforming subjects (such as a person's face or a moving object) from monocular or multi-view video rather than only static, multi-view-photographed scenes.

## Computational Considerations

The original NeRF formulation is computationally expensive both to train and to render, because computing a single rendered pixel requires evaluating the underlying neural network at dozens of sampled points along the corresponding ray, and rendering a full image or training on a full batch of rays therefore requires an enormous number of individual network evaluations, a cost that scales with both image resolution and the number of samples used per ray, and which historically made original-formulation NeRF training a process taking many hours to a day or more on a single GPU for a single scene.

Acceleration techniques fall into several broad categories: reducing the number of samples needed per ray through more efficient sampling strategies (such as proposal networks that predict where density is likely to be concentrated before the final, more expensive network evaluation), replacing or augmenting the implicit neural representation with explicit or hybrid spatial data structures (voxel grids, hash grids, or tensor decompositions) that are faster to query than a deep multilayer perceptron, and, in the case of 3D Gaussian Splatting, replacing ray-marching-based rendering entirely with a rasterization-style splatting pipeline that leverages highly optimized graphics hardware rasterization rather than per-sample neural network inference.

Memory considerations differ substantially across representations: a compact implicit NeRF network may require only a few megabytes to represent an entire scene (since the scene's appearance and geometry are compressed into the network's weights), while explicit representations such as voxel grids, hash grids, or Gaussian Splatting's collection of primitives can require substantially more memory (particularly Gaussian Splatting scenes reconstructed at high fidelity, which can involve millions of individual Gaussian primitives, each with its own learned parameters), representing a direct trade-off between rendering speed and memory footprint that practitioners must consider based on their deployment target, whether a memory-constrained mobile device or a well-resourced rendering server.

## Practical Implementation Strategies

A practical NeRF or Gaussian Splatting reconstruction pipeline begins with capturing a sufficient number of photographs of the target scene from a wide range of viewpoints with adequate overlap between adjacent views (insufficient view coverage or overlap leads to poorly reconstructed or entirely missing geometry in under-observed regions), followed by running structure-from-motion software to recover camera poses and, commonly, an initial sparse 3D point cloud used to initialize Gaussian Splatting's primitives (NeRF, by contrast, typically initializes its implicit representation without requiring this sparse point cloud, though it still requires the recovered camera poses).

Choosing between an implicit NeRF-style representation and an explicit Gaussian-Splatting-style representation depends heavily on the deployment requirements: applications requiring real-time or interactive rendering rates (game engines, live virtual production, VR experiences) generally favor Gaussian Splatting or other explicit, rasterization-friendly representations, while applications prioritizing minimal memory footprint, smooth and well-behaved geometry for downstream editing or physical simulation, or the flexibility of a continuous, resolution-independent representation may still favor an implicit neural formulation.

Handling scenes with challenging conditions, including reflective or transparent surfaces, inconsistent lighting across captured photographs (such as photographs captured at different times of day, a common issue for outdoor or tourist-landmark reconstructions built from internet photo collections), and dynamic elements like moving people or vehicles present during capture, generally requires specialized extensions beyond the base NeRF or Gaussian Splatting formulation, such as appearance embeddings that let the model account for per-image lighting or exposure variation, or explicit modeling and masking of transient, non-static scene content.

## Benchmark Datasets & Evaluation

The original NeRF paper's synthetic Blender dataset, consisting of renders of several complex synthetic objects from known camera poses with ground-truth held-out views, remains a standard controlled benchmark for measuring reconstruction fidelity in a setting free of the pose-estimation and lighting-consistency noise present in real-world captures. The LLFF (Local Light Field Fusion) dataset and the more recent Mip-NeRF 360 dataset provide real-world, forward-facing and full 360-degree captured scenes respectively, used to evaluate performance under realistic capture conditions, including the more challenging case of unbounded, full-surround scenes where background content extends to effectively infinite distance from the camera.

Standard image quality metrics used to evaluate rendered held-out views against ground-truth photographs include PSNR (peak signal-to-noise ratio, a straightforward pixel-wise error metric), SSIM (structural similarity index, which better captures perceptual structural similarity than raw pixel error), and LPIPS (Learned Perceptual Image Patch Similarity, a metric based on deep network feature distances that correlates more closely with human perceptual judgments of image similarity than either PSNR or SSIM alone), with most published NeRF and Gaussian Splatting research reporting all three metrics together to give a fuller picture of reconstruction quality that is not overly sensitive to any single metric's particular blind spots.

Training and rendering speed, typically reported as training time to reach a given quality threshold and achieved frames-per-second at rendering time, have become increasingly central evaluation criteria alongside pure visual fidelity, reflecting the field's substantial recent focus on efficiency improvements (Instant-NGP-style training in seconds, Gaussian Splatting rendering in real time) rather than purely on maximizing reconstruction quality without regard to computational cost.

## Key Challenges & Limitations

Both NeRF and Gaussian Splatting representations are fundamentally scene-specific: the standard formulation optimizes a new representation from scratch for each individual scene, using only the photographs of that particular scene, meaning the trained representation does not generalize to novel scenes and must be retrained (or, for faster variants, re-optimized from a reasonable initialization) for every new environment to be captured, in contrast to a general-purpose image or video model that, once trained, applies to arbitrary new inputs without per-instance optimization.

Reconstructing regions of a scene that are poorly observed in the input photographs (occluded areas, surfaces only glimpsed from a narrow range of angles, or regions entirely outside the captured viewpoint coverage) reliably produces artifacts, characteristically appearing as blurry, semi-transparent "floater" geometry or incorrect density hallucinated by the optimization process attempting to explain the limited available observations, a persistent challenge given that both representations are fit purely to reproduce the observed training photographs without any strong external prior about what unobserved scene regions should look like.

View-dependent and non-Lambertian effects, including strong specular reflections, transparency, and complex subsurface scattering, while better handled by radiance-field-style representations than by older purely geometric approaches, still present reconstruction difficulty, particularly for Gaussian Splatting's relatively low-order spherical-harmonic representation of view-dependent color, which can struggle to capture very sharp, highly directional specular highlights compared to a sufficiently expressive implicit neural network.

## Hyperparameter Tuning

The number of samples taken per ray, and the strategy used to distribute those samples along the ray (uniform sampling, coarse-to-fine hierarchical sampling as in the original NeRF, or learned proposal-network-guided sampling), directly trades off rendering and training speed against the ability to accurately capture fine, thin, or high-frequency geometric detail, with too few samples producing blurry or missing fine structure and too many samples imposing an often prohibitive computational cost.

The maximum frequency used in positional encoding (or, for hash-grid-based methods, the number of resolution levels and the hash table size per level) directly controls the finest spatial detail the representation can capture, and setting this too low produces overly smooth, detail-lacking reconstructions, while setting it too high, particularly relative to the amount and quality of available training views, can allow the representation to overfit to view-specific noise or camera calibration error rather than genuine scene detail.

For Gaussian Splatting specifically, the densification and pruning schedule, controlling when and how new Gaussian primitives are added in under-reconstructed regions and when low-opacity or otherwise unhelpful primitives are removed during training, substantially affects both the final reconstruction quality and the total number of primitives (and thus memory footprint and rendering speed) of the resulting scene representation, representing a quality-versus-efficiency trade-off that is actively tuned in most practical Gaussian Splatting pipelines.

## Real-World Applications & Case Studies

Visual effects and virtual production studios have adopted NeRF and Gaussian Splatting techniques to capture real filming locations or physical sets as fully explorable 3D scenes, enabling virtual camera moves and compositing shots that would be impractical or impossible to achieve with the physical camera during the original shoot, effectively turning a real location into reusable, photorealistic 3D asset.

E-commerce platforms have used these techniques to generate interactive, photorealistic 3D product visualizations directly from a small number of product photographs, letting customers view and rotate products from arbitrary angles without requiring dedicated 3D modeling work by an artist, substantially reducing the cost of producing rich, interactive product presentation content at scale.

Autonomous driving and robotics research groups have explored NeRF-style and Gaussian-Splatting-style scene representations as a component of simulation pipelines, using captured real-world driving scenes reconstructed as radiance fields or Gaussian primitive collections to generate realistic synthetic sensor data (camera images from novel simulated vehicle trajectories) for testing and training perception systems under scenarios and viewpoints not present in the original captured data.

## Integration with Other Methods

Diffusion models have been combined with NeRF-style representations in text-to-3D and image-to-3D generation pipelines, most notably through score distillation sampling, in which a pretrained 2D text-to-image diffusion model is used as a learned prior to guide the optimization of a NeRF (or Gaussian Splatting) representation toward matching a text prompt or reference image, without requiring any 3D training data at all, effectively repurposing a 2D generative prior to synthesize entirely new 3D content, a substantially different and more constrained problem than reconstructing an existing physical scene from real photographs.

Semantic segmentation and other 2D scene-understanding techniques have been integrated into radiance field representations by distilling 2D semantic feature predictions (from pretrained image segmentation or vision-language models) into an auxiliary output of the radiance field itself, producing 3D-consistent semantic or language-queryable scene representations that support tasks like open-vocabulary 3D object localization directly within the reconstructed scene.

Simultaneous localization and mapping (SLAM) systems in robotics have begun incorporating NeRF-style and Gaussian-Splatting-style scene representations as their underlying map representation, replacing traditional point-cloud or voxel-grid maps with a differentiable radiance-field-based map that can be incrementally updated as a robot explores an environment while simultaneously being useful for photorealistic rendering and downstream planning tasks.

## Future Research Directions

Improving generalization across scenes, so that a single trained model can represent or quickly adapt to new scenes given only a handful of input images (rather than requiring lengthy per-scene optimization from scratch), remains an active research direction, with generalizable and feed-forward radiance-field architectures aiming to amortize much of the reconstruction cost through a shared, pretrained network applicable across many different scenes.

Improving robustness to imperfect or entirely unknown camera poses, reducing or eliminating the dependency on accurate structure-from-motion preprocessing, is an active area given that pose estimation failures are a common practical bottleneck for reconstructing scenes with limited or low-texture visual content, where structure-from-motion algorithms themselves frequently struggle to recover accurate camera poses.

Extending real-time, high-fidelity reconstruction and rendering to large-scale, city-scale, or continuously updating dynamic environments, combining the efficiency advances of methods like Gaussian Splatting with robust handling of dynamic content, lighting variation, and enormous spatial scale, represents a substantial ongoing engineering and research challenge as these techniques move from controlled, bounded-scene demonstrations toward large-scale, real-world deployment.

## Summary & Key Takeaways

Neural Radiance Fields represent a 3D scene implicitly as a continuous, neural-network-parameterized function mapping spatial position and viewing direction to color and density, rendered via differentiable volume rendering, enabling photorealistic novel view synthesis directly optimized from a set of posed photographs without requiring an explicit 3D geometric reconstruction step.

Positional encoding (or, in accelerated variants, learned multi-resolution hash-grid features) is essential to overcoming the spectral bias of coordinate-based neural networks toward smooth, low-frequency functions, enabling the representation of fine spatial and appearance detail.

3D Gaussian Splatting offers an explicit, rasterization-friendly alternative representation achieving comparable visual fidelity with substantially faster real-time rendering, at the cost of a typically larger memory footprint from storing millions of explicit Gaussian primitives, illustrating a broader implicit-versus-explicit representational trade-off that continues to shape research in this area.

Key remaining challenges include per-scene optimization cost and lack of cross-scene generalization, artifacts in poorly observed scene regions, and robust handling of dynamic, non-Lambertian, and large-scale environments, all active areas of ongoing research as radiance-field-based techniques move toward broader practical deployment across visual effects, e-commerce, robotics, and generative 3D content creation.

Keywords: neural radiance fields, NeRF, novel view synthesis, volume rendering, positional encoding, 3D Gaussian Splatting, Instant-NGP, hash grid encoding, structure from motion, COLMAP, view-dependent rendering, spherical harmonics, score distillation sampling, text-to-3D generation, dynamic NeRF, deformable radiance fields, PSNR SSIM LPIPS, volumetric compositing, real-time rendering, differentiable rendering

---

## Appendix: Practical Labs

### Lab 1: Volume Rendering — Discrete Ray Compositing from Sampled Density and Color

import numpy as np

np.random.seed(0)


def discrete_volume_render(sigmas, colors, deltas):
    """
    Implements the discrete volume rendering approximation used by NeRF:
        C_hat = sum_i T_i * (1 - exp(-sigma_i * delta_i)) * c_i
        T_i = exp(-sum_{j<i} sigma_j * delta_j)

    sigmas: (N,) array of per-sample volume densities along one ray
    colors: (N, 3) array of per-sample RGB colors along the ray
    deltas: (N,) array of distances between consecutive samples
    """
    alpha = 1.0 - np.exp(-sigmas * deltas)  # per-sample "opacity"
    # Transmittance T_i requires the *cumulative* product of (1 - alpha_j) for j < i
    transmittance = np.concatenate([[1.0], np.cumprod(1.0 - alpha + 1e-10)[:-1]])
    weights = transmittance * alpha  # shape (N,)
    rendered_color = np.sum(weights[:, None] * colors, axis=0)
    rendered_depth = np.sum(weights * np.cumsum(deltas))
    accumulated_opacity = np.sum(weights)
    return rendered_color, rendered_depth, accumulated_opacity, weights


def test_volume_rendering_basic_properties():
    n_samples = 32

    # Case 1: a single, fully opaque surface roughly in the middle of the ray
    sigmas = np.zeros(n_samples)
    sigmas[15] = 50.0  # very high density = essentially opaque at this sample
    deltas = np.full(n_samples, 0.1)
    colors = np.zeros((n_samples, 3))
    colors[15] = np.array([1.0, 0.0, 0.0])  # red surface

    rendered_color, rendered_depth, opacity, weights = discrete_volume_render(
        sigmas, colors, deltas
    )

    print(f"Rendered color (should be near red): {rendered_color}")
    print(f"Accumulated opacity (should be near 1.0): {opacity:.4f}")
    print(f"Weight concentration at sample 15: {weights[15]:.4f} "
          f"(sum of all weights: {weights.sum():.4f})")

    assert opacity > 0.95, "A single very high-density sample should nearly fully occlude the ray"
    assert rendered_color[0] > 0.9, "Rendered color should be dominated by the red surface"
    assert weights[15] > 0.9, "Almost all rendering weight should concentrate at the opaque sample"

    # Case 2: fully empty ray (zero density everywhere) should render as black / zero opacity
    empty_sigmas = np.zeros(n_samples)
    empty_colors = np.random.rand(n_samples, 3)  # colors shouldn't matter if density is 0
    empty_color_out, _, empty_opacity, _ = discrete_volume_render(empty_sigmas, empty_colors, deltas)

    print(f"Empty ray opacity (should be ~0): {empty_opacity:.6f}")
    assert empty_opacity < 1e-4, "A ray through empty space should have near-zero accumulated opacity"

    print("Volume rendering basic properties test passed.")


if __name__ == "__main__":
    test_volume_rendering_basic_properties()

### Lab 2: Positional Encoding and Its Effect on Representable Frequency Content

import numpy as np

np.random.seed(1)


def positional_encoding(x, num_frequencies):
    """Maps a scalar (or array of scalars) x to [sin(2^0 pi x), cos(2^0 pi x), ...,
    sin(2^{L-1} pi x), cos(2^{L-1} pi x)], the encoding used in the original NeRF paper."""
    x = np.atleast_1d(x)
    encoded = []
    for i in range(num_frequencies):
        freq = (2.0 ** i) * np.pi
        encoded.append(np.sin(freq * x))
        encoded.append(np.cos(freq * x))
    return np.stack(encoded, axis=-1)  # shape (..., 2 * num_frequencies)


class TinyMLP:
    """A minimal 2-layer MLP for regression, trained via simple gradient descent,
    used to demonstrate spectral bias with and without positional encoding."""

    def __init__(self, in_dim, hidden_dim=32, seed=0):
        rng = np.random.RandomState(seed)
        self.W1 = rng.randn(in_dim, hidden_dim) * np.sqrt(2.0 / in_dim)
        self.b1 = np.zeros(hidden_dim)
        self.W2 = rng.randn(hidden_dim, 1) * np.sqrt(2.0 / hidden_dim)
        self.b2 = np.zeros(1)

    def forward(self, x):
        h = np.maximum(0, x @ self.W1 + self.b1)  # ReLU
        return (h @ self.W2 + self.b2).squeeze(-1), h

    def train_step(self, x, y_true, lr=0.01):
        y_pred, h = self.forward(x)
        error = y_pred - y_true
        loss = np.mean(error ** 2)

        grad_out = (2.0 * error / len(y_true))[:, None]
        grad_W2 = h.T @ grad_out
        grad_b2 = grad_out.sum(axis=0)

        grad_h = grad_out @ self.W2.T
        grad_h[h <= 0] = 0  # ReLU derivative
        grad_W1 = x.T @ grad_h
        grad_b1 = grad_h.sum(axis=0)

        self.W1 -= lr * grad_W1
        self.b1 -= lr * grad_b1
        self.W2 -= lr * grad_W2
        self.b2 -= lr * grad_b2
        return loss


def fit_high_frequency_signal(use_encoding, num_frequencies=6, n_steps=3000):
    x_raw = np.linspace(0, 1, 200)
    # A high-frequency target signal that a smooth-biased MLP will struggle to fit directly
    y_true = np.sin(2 * np.pi * 8 * x_raw)

    if use_encoding:
        x_input = positional_encoding(x_raw, num_frequencies)
    else:
        x_input = x_raw[:, None]

    model = TinyMLP(in_dim=x_input.shape[1], hidden_dim=64, seed=3)
    losses = []
    for step in range(n_steps):
        loss = model.train_step(x_input, y_true, lr=0.05)
        losses.append(loss)

    final_pred, _ = model.forward(x_input)
    return losses, final_pred, y_true


def test_positional_encoding_improves_high_frequency_fit():
    losses_no_enc, pred_no_enc, y_true = fit_high_frequency_signal(use_encoding=False)
    losses_with_enc, pred_with_enc, _ = fit_high_frequency_signal(use_encoding=True)

    final_loss_no_enc = losses_no_enc[-1]
    final_loss_with_enc = losses_with_enc[-1]

    print(f"Final training loss WITHOUT positional encoding: {final_loss_no_enc:.5f}")
    print(f"Final training loss WITH positional encoding:    {final_loss_with_enc:.5f}")

    assert final_loss_with_enc < final_loss_no_enc, (
        "Positional encoding should allow the MLP to fit the high-frequency "
        "target signal substantially better than raw coordinates"
    )
    assert final_loss_with_enc < 0.05, "Encoded MLP should fit the high-frequency signal reasonably well"

    print("Positional encoding spectral-bias test passed.")


if __name__ == "__main__":
    test_positional_encoding_improves_high_frequency_fit()

### Lab 3: Ray-AABB Intersection for Bounding-Box-Restricted Sampling

import numpy as np

np.random.seed(2)


def ray_aabb_intersect(ray_origin, ray_direction, box_min, box_max):
    """Computes the near and far intersection distances of a ray with an
    axis-aligned bounding box (AABB), used in practice to restrict NeRF
    ray sampling to the region of space actually containing the scene,
    rather than wastefully sampling empty space far outside the scene bounds."""
    ray_direction = np.where(np.abs(ray_direction) < 1e-10, 1e-10, ray_direction)
    t_min = (box_min - ray_origin) / ray_direction
    t_max = (box_max - ray_origin) / ray_direction

    t1 = np.minimum(t_min, t_max)
    t2 = np.maximum(t_min, t_max)

    t_near = np.max(t1)
    t_far = np.min(t2)

    hit = t_far >= max(t_near, 0.0)
    return hit, t_near, t_far


def generate_ray_samples_within_bounds(ray_origin, ray_direction, box_min, box_max, n_samples):
    hit, t_near, t_far = ray_aabb_intersect(ray_origin, ray_direction, box_min, box_max)
    if not hit:
        return None, None

    t_near = max(t_near, 0.0)
    t_values = np.linspace(t_near, t_far, n_samples)
    points = ray_origin[None, :] + t_values[:, None] * ray_direction[None, :]
    return t_values, points


def test_ray_aabb_intersection():
    box_min = np.array([-1.0, -1.0, -1.0])
    box_max = np.array([1.0, 1.0, 1.0])

    # Ray that passes directly through the box, along the x-axis
    ray_origin = np.array([-5.0, 0.0, 0.0])
    ray_direction = np.array([1.0, 0.0, 0.0])

    hit, t_near, t_far = ray_aabb_intersect(ray_origin, ray_direction, box_min, box_max)
    print(f"Ray through box: hit={hit}, t_near={t_near:.2f}, t_far={t_far:.2f}")

    assert hit, "A ray aimed directly through the box should register a hit"
    assert abs(t_near - 4.0) < 1e-6, "Entry point should be at distance 4 (from x=-5 to x=-1)"
    assert abs(t_far - 6.0) < 1e-6, "Exit point should be at distance 6 (from x=-5 to x=1)"

    t_values, points = generate_ray_samples_within_bounds(
        ray_origin, ray_direction, box_min, box_max, n_samples=10
    )
    assert points is not None, "Samples should be generated for a ray that hits the box"
    assert np.all(points[:, 0] >= -1.0 - 1e-6) and np.all(points[:, 0] <= 1.0 + 1e-6), (
        "All sampled points should lie within the box's x-extent"
    )

    # Ray that misses the box entirely
    miss_ray_origin = np.array([-5.0, 5.0, 5.0])
    miss_direction = np.array([1.0, 0.0, 0.0])
    hit_miss, _, _ = ray_aabb_intersect(miss_ray_origin, miss_direction, box_min, box_max)
    print(f"Ray missing box: hit={hit_miss}")
    assert not hit_miss, "A ray that does not pass through the box should not register a hit"

    print("Ray-AABB intersection test passed.")


if __name__ == "__main__":
    test_ray_aabb_intersection()

### Lab 4: 2D Gaussian Splat Rasterization — Projecting and Alpha-Compositing Elliptical Primitives

import numpy as np

np.random.seed(3)


def gaussian_2d_footprint(pixel_coords, mean, covariance):
    """Evaluates a 2D Gaussian's density at a set of pixel coordinates,
    used to compute each Gaussian primitive's contribution ("splat") to nearby pixels."""
    inv_cov = np.linalg.inv(covariance)
    diff = pixel_coords - mean  # shape (n_pixels, 2)
    exponent = -0.5 * np.einsum('ij,jk,ik->i', diff, inv_cov, diff)
    norm_factor = 1.0 / (2 * np.pi * np.sqrt(np.linalg.det(covariance)))
    return norm_factor * np.exp(exponent)


def alpha_composite_splats(splats, image_shape):
    """
    splats: list of dicts with keys 'mean' (2,), 'covariance' (2,2), 'color' (3,),
            'opacity' (scalar), sorted from FRONT (closest to camera) to BACK.
    Renders by front-to-back alpha compositing, matching the order-dependent
    accumulation used in Gaussian Splatting rasterization.
    """
    h, w = image_shape
    yy, xx = np.meshgrid(np.arange(h), np.arange(w), indexing='ij')
    pixel_coords = np.stack([xx.ravel(), yy.ravel()], axis=-1).astype(float)

    accumulated_color = np.zeros((h * w, 3))
    accumulated_alpha = np.zeros(h * w)

    for splat in splats:
        density = gaussian_2d_footprint(pixel_coords, splat['mean'], splat['covariance'])
        density = density / (density.max() + 1e-10)  # normalize footprint to [0, 1] peak
        splat_alpha = splat['opacity'] * density

        # Front-to-back compositing: contribution is weighted by remaining "unfilled" transmittance
        remaining_transmittance = 1.0 - accumulated_alpha
        contribution = remaining_transmittance * splat_alpha
        accumulated_color += contribution[:, None] * splat['color'][None, :]
        accumulated_alpha += contribution

    return accumulated_color.reshape(h, w, 3), accumulated_alpha.reshape(h, w)


def test_gaussian_splat_compositing():
    image_shape = (20, 20)

    # A red splat in front, and a blue splat behind it at the same location
    splats = [
        {
            'mean': np.array([10.0, 10.0]),
            'covariance': np.array([[4.0, 0.0], [0.0, 4.0]]),
            'color': np.array([1.0, 0.0, 0.0]),
            'opacity': 0.9,
        },
        {
            'mean': np.array([10.0, 10.0]),
            'covariance': np.array([[6.0, 0.0], [0.0, 6.0]]),
            'color': np.array([0.0, 0.0, 1.0]),
            'opacity': 0.9,
        },
    ]

    rendered_image, alpha_map = alpha_composite_splats(splats, image_shape)

    center_pixel_color = rendered_image[10, 10]
    print(f"Rendered color at splat center: {center_pixel_color}")
    print(f"Accumulated alpha at splat center: {alpha_map[10, 10]:.4f}")

    # The front (red) splat should dominate the color at the exact center, since it
    # occludes most of the back (blue) splat's contribution there
    assert center_pixel_color[0] > center_pixel_color[2], (
        "Front red splat should dominate over the occluded back blue splat at the center"
    )
    assert alpha_map[10, 10] > 0.8, "Accumulated opacity at the splat center should be high"

    # Far from both splats, the pixel should receive almost no contribution
    corner_alpha = alpha_map[0, 0]
    print(f"Accumulated alpha at image corner (far from splats): {corner_alpha:.6f}")
    assert corner_alpha < 0.05, "Pixels far from any splat should have near-zero accumulated alpha"

    print("Gaussian splat alpha-compositing test passed.")


if __name__ == "__main__":
    test_gaussian_splat_compositing()