3D Deep Learning and Point Cloud Processing is the neural network discipline that processes three-dimensional geometric data — point clouds from LiDAR sensors, depth cameras, and 3D scanners — for object detection, segmentation, and scene understanding in autonomous driving, robotics, and industrial inspection, where the unstructured, sparse, and orderless nature of 3D point data requires specialized architectures fundamentally different from 2D image processing.

Point Cloud Data Structure

A point cloud is a set of N points {(x_i, y_i, z_i, f_i)} where (x, y, z) are 3D coordinates and f_i are optional features (intensity, RGB color, surface normals). Key properties:

• Unstructured: No grid or connectivity information. Points are scattered irregularly in 3D space.
• Permutation Invariant: The point set {A, B, C} is the same as {C, A, B} — the network must be invariant to input ordering.
• Sparse: In outdoor LiDAR, 99%+ of the 3D volume is empty. A typical LiDAR frame: 100,000-300,000 points in a 100m × 100m × 10m volume.

Point-Based Architectures

• PointNet (2017): The foundational architecture. Processes each point independently with shared MLPs, then applies a max-pool (symmetric function) to achieve permutation invariance. Global feature captures the overall shape. Limitation: no local structure — each point is processed in isolation.
• PointNet++: Hierarchical PointNet. Uses farthest-point sampling and ball query to group local neighborhoods, applies PointNet within each group, then progressively aggregates. Captures multi-scale local geometry.
• Point Transformer: Applies self-attention to local point neighborhoods. Vector attention (not scalar) captures directional relationships between points. State-of-the-art on indoor segmentation (S3DIS, ScanNet).

Voxel-Based Architectures

• VoxelNet: Divides 3D space into regular voxels, aggregates points within each voxel using PointNet, then applies 3D convolutions on the voxel grid. Combines the regularity of grids with point-level features.
• SECOND (Spatially Efficient Convolution): Uses 3D sparse convolutions — only computes on occupied voxels, skipping empty space. 10-100x faster than dense 3D convolution.
• CenterPoint: Voxel-based 3D object detection. After sparse 3D convolution, the BEV (Bird's Eye View) feature map is processed by a 2D detection head that predicts object centers, sizes, and orientations. The dominant architecture for LiDAR-based autonomous driving detection.

Autonomous Driving Pipeline

1. LiDAR Point Cloud (64-128 beams, 10-20 Hz, 100K+ points/frame). 2. 3D Detection: CenterPoint/PointPillars detects vehicles, pedestrians, cyclists with 3D bounding boxes (x, y, z, w, h, l, yaw). 3. Multi-Frame Fusion: Accumulate multiple LiDAR sweeps and ego-motion compensate for denser point clouds and temporal consistency. 4. Camera-LiDAR Fusion: Project 3D features onto 2D images or lift 2D features to 3D (BEVFusion) for complementary modality fusion.

3D Deep Learning is the perception technology that gives machines spatial understanding of the physical world — processing the raw 3D geometry captured by range sensors into the object-level scene descriptions that autonomous vehicles and robots need to navigate and interact safely.