Point Cloud Deep Learning is the family of neural network architectures that process raw 3D point clouds (unordered sets of XYZ coordinates with optional features like color, intensity, or normals) for tasks including 3D object classification, semantic segmentation, and object detection — addressing the fundamental challenge that point clouds are unordered, irregular, and sparse, requiring architectures invariant to point permutation and robust to density variation, unlike the regular grid structure that enables standard CNNs on images.

The Point Cloud Challenge

A LiDAR scan or depth sensor produces {(x₁,y₁,z₁), (x₂,y₂,z₂), ...} — an unordered set of 3D points. Unlike pixels on a regular 2D grid, points have no canonical ordering, variable density (more points on nearby objects), and no natural neighborhood structure for convolution.

PointNet (Qi et al., 2017)

The pioneering architecture for direct point cloud processing:

• Per-Point MLP: Each point's (x,y,z) is independently processed through shared MLPs (64→128→1024 dimensions).
• Symmetric Aggregation: Max-pooling across all points produces a global feature vector. Max-pooling is permutation-invariant — solves the ordering problem.
• Classification: Global feature → FC layers → class scores.
• Segmentation: Concatenate per-point features with global feature → per-point MLP → per-point class scores.
• Limitation: No local structure — max-pooling over all points ignores spatial neighborhoods. Cannot capture local geometric patterns (edges, corners, planes).

PointNet++ (Qi et al., 2017)

Hierarchical point set learning:

• Set Abstraction Layers: (1) Farthest-point sampling selects representative centroids. (2) Ball query groups neighboring points around each centroid. (3) PointNet applied to each local group produces a per-centroid feature. Repeated for multiple levels — like CNN pooling hierarchy but for irregular point sets.
• Multi-Scale Grouping: Use multiple ball radii at each level to capture features at different scales — handles variable density.

3D Sparse Convolution

For voxelized point clouds (discretize 3D space into regular voxels):

• Minkowski Engine / SpConv: Sparse convolution operates only on occupied voxels — avoids computation on the 99%+ empty voxels. Hash-table-based indexing for sparse data.
• Efficiency: An indoor scene with 100K points in a 256³ voxel grid: 99.97% of voxels are empty. Dense 3D convolution would process 16.7M voxels. Sparse convolution processes only ~100K — 167× more efficient.

Transformer-Based

• Point Transformer: Self-attention with learnable positional encoding applied to local neighborhoods. Attention weights capture the relative importance of neighboring points.
• Stratified Transformer: Stratified sampling strategy for more effective long-range attention in point clouds.

Detection in 3D

• VoxelNet / SECOND: Voxelize LiDAR point cloud → sparse 3D convolution → 2D BEV (bird's-eye view) feature map → 2D detection head. Standard for autonomous driving.
• CenterPoint: Detect objects as center points in the BEV feature map, then refine 3D bounding boxes including height and orientation.

Point Cloud Deep Learning is the 3D perception technology that enables machines to understand the physical world from sensor data — processing the raw geometric measurements from LiDAR, depth cameras, and photogrammetry into the semantic understanding required for autonomous driving, robotics, and 3D scene understanding.