sensor fusion
**Sensor fusion combines measurements from multiple sensors or times into a more complete, accurate, and robust state estimate.** Cameras provide semantics, lidar provides geometry, radar provides range and velocity, IMUs provide high-rate motion, GNSS provides global reference, and fusion compensates for their complementary weaknesses. Fusion requires coordinate frames, timestamps, uncertainty, latency, observability, correlation, failure modes, and target state. More sensors do not guarantee better results: a biased or miscalibrated source can make a confident fused estimate worse. A production perception claim specifies the sensor, scene distribution, label ontology, spatial and temporal resolution, operating range, latency deadline, target hardware, confidence policy, and consequence of a miss or false alarm. Dataset accuracy alone is insufficient when lighting, weather, motion, occlusion, calibration, geography, demographics, and sensor aging differ from the benchmark.
**Architecture, representation, and operating mechanism.** Early fusion combines raw or minimally processed data, mid-level fusion aligns learned features, late fusion combines independent detections or decisions, and state-estimation fusion uses Kalman, information, particle, factor-graph, or optimization methods. Hybrid stacks often use several levels. Sensors are calibrated and time-aligned, transformed into a common representation, associated with tracks or landmarks, weighted by uncertainty, and updated into a fused state. Gating rejects implausible associations; health monitors reduce or remove a failed sensor. Position/orientation/velocity error, detection and tracking quality, consistency, covariance calibration, availability, time-to-detect faults, recovery, latency, synchronization tolerance, compute, bandwidth, power, and performance under sensor dropout matter. Cameras, lidar, radar, IMUs, optics, illumination, clocks, mounts, compute, memory, interconnect, thermal limits, middleware, trackers, maps, planning, UI, and human escalation form one system. A faster neural network may not reduce end-to-end latency if decode, transfer, synchronization, or postprocessing dominates. Evaluation reports task quality, calibration, subgroup and condition slices, robustness, tail latency, throughput, memory, power, model size, preprocessing and postprocessing cost, and uncertainty across runs. Leakage-resistant splits separate locations, subjects, devices, and time where needed; confidence intervals and error taxonomies expose whether a headline score represents deployable behavior.
**Implementation, hardware, and failure modes.** Intrinsic/extrinsic calibration, PTP or hardware timestamps, rolling-shutter compensation, interpolation, data association, occupancy/BEV grids, cross-attention, Kalman/UKF/EKF, factor graphs, learned uncertainty, out-of-sequence updates, and redundancy management shape design. Multiple high-rate sensors stress I/O, memory, timestamp hardware, DMA, image/lidar accelerators, and interconnect. Central fusion maximizes joint context; distributed fusion reduces bandwidth but risks information loss and correlated estimates. Clock offset looks like spatial error, extrinsic drift causes ghost objects, weather degrades camera/lidar differently, radar multipath creates false targets, GNSS can be blocked or spoofed, correlated errors are double-counted, and learned fusion may ignore a modality. Engineering must include data movement, finite precision, resource contention, numerical or physical limits, error propagation, and deterministic behavior when assumptions are violated. The pipeline includes sensing, synchronization, calibration, ingestion, annotation, augmentation, training, evaluation, compilation, quantization, serving, monitoring, feedback, rollback, and dataset/model retirement. Raw data, labels, ontology versions, transforms, checkpoints, compiler artifacts, thresholds, and hardware profiles are traceable so a field failure can be reproduced.
**Evaluation, verification, and deployment.** Inject timestamp and calibration errors, dropout, stuck and biased sensors, weather and lighting, spoofing, dynamic occlusion, high acceleration, bandwidth loss, and compute overload; check covariance consistency and graceful degradation as well as nominal accuracy. Fusion feeds localization, mapping, perception, prediction, control, AR rendering, and safety monitors. Frame conventions, calibration storage, health state, fallback behavior, and diagnostic visibility are interfaces, not implementation details. Camera, microphone, location, and biometric sensors have different privacy obligations. Minimize modalities and retention, isolate raw data, document purposes, secure calibration and firmware, and provide user or operator controls. Verification combines held-out and out-of-distribution sets, synthetic stress with real validation, adversarial and corruption tests, calibration analysis, edge-case replay, hardware-in-the-loop timing, long-duration soak, human review, and shadow or canary deployment. Failures feed collection and labeling rather than being hidden by aggregate averages. The pipeline includes sensing, synchronization, calibration, ingestion, annotation, augmentation, training, evaluation, compilation, quantization, serving, monitoring, feedback, rollback, and dataset/model retirement. Raw data, labels, ontology versions, transforms, checkpoints, compiler artifacts, thresholds, and hardware profiles are traceable so a field failure can be reproduced. Evaluation reports task quality, calibration, subgroup and condition slices, robustness, tail latency, throughput, memory, power, model size, preprocessing and postprocessing cost, and uncertainty across runs. Leakage-resistant splits separate locations, subjects, devices, and time where needed; confidence intervals and error taxonomies expose whether a headline score represents deployable behavior.
| Fusion level | Combined data | Strength | Limitation | Best fit |
|---|---|---|---|---|
| Early/raw | Measurements or dense grids | Maximum information interaction | Alignment/bandwidth/compute | Tightly synchronized sensors |
| Mid/feature | Learned feature maps | Strong semantic complementarity | Training and interpretability | Deep perception |
| Late/decision | Objects/scores/tracks | Modular and fault isolating | Information already discarded | Heterogeneous subsystems |
| State estimator | State + covariance | Principled dynamics/uncertainty | Model and association assumptions | Navigation/tracking |
| Factor graph | Measurements over time | Global smoothing and loop constraints | Optimization latency/complexity | SLAM and mapping |
```svg
```
**Selection and practical application.** Choose early fusion when alignment and bandwidth support rich interaction, mid-level fusion for learned complementarity, late fusion for modular fault isolation, and probabilistic state fusion for interpretable uncertainty and dynamics. Autonomous vehicles, robots, drones, smartphones, AR/VR, navigation, industrial monitoring, medical devices, tracking, and smart infrastructure depend on multisensor estimates. Cameras, lidar, radar, IMUs, optics, illumination, clocks, mounts, compute, memory, interconnect, thermal limits, middleware, trackers, maps, planning, UI, and human escalation form one system. A faster neural network may not reduce end-to-end latency if decode, transfer, synchronization, or postprocessing dominates. A production perception claim specifies the sensor, scene distribution, label ontology, spatial and temporal resolution, operating range, latency deadline, target hardware, confidence policy, and consequence of a miss or false alarm. Dataset accuracy alone is insufficient when lighting, weather, motion, occlusion, calibration, geography, demographics, and sensor aging differ from the benchmark. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.