pose estimation
**Pose estimation localizes anatomical, robotic, or object keypoints and their spatial relationships in images or video.** Human pose supports sports, fitness, ergonomics, sign language, animation, AR/VR, safety, medical analysis, and human-robot interaction; object pose supports manipulation and inspection. 2D pose predicts image keypoints, 3D pose estimates coordinates relative to camera or body, and six-degree-of-freedom object pose estimates translation and rotation. Visibility, skeleton topology, coordinate frame, multi-person association, and uncertainty must be specified. A production perception claim specifies the sensor, scene distribution, label ontology, spatial and temporal resolution, operating range, latency deadline, target hardware, confidence policy, and consequence of a miss or false alarm. Dataset accuracy alone is insufficient when lighting, weather, motion, occlusion, calibration, geography, demographics, and sensor aging differ from the benchmark.
**Architecture, representation, and operating mechanism.** Top-down systems detect each person then run a keypoint model; bottom-up systems detect joints globally then group them; single-stage methods regress people and poses jointly. OpenPose uses part-affinity fields, HRNet preserves high resolution, ViTPose uses Transformer features, and MediaPipe emphasizes efficient tracking. Models output heatmaps, coordinate distributions, offsets, or direct coordinates. Decoding finds peaks and refines locations; association links joints to individuals; temporal filtering or tracking stabilizes sequences; 3D lifting uses camera geometry and learned priors. PCK, OKS-based average precision, MPJPE for 3D, angular error, visibility accuracy, identity switches, jitter, temporal lag, multi-person recall, latency, FPS, power, and behavior under occlusion and truncation matter. Cameras, lidar, radar, IMUs, optics, illumination, clocks, mounts, compute, memory, interconnect, thermal limits, middleware, trackers, maps, planning, UI, and human escalation form one system. A faster neural network may not reduce end-to-end latency if decode, transfer, synchronization, or postprocessing dominates. Evaluation reports task quality, calibration, subgroup and condition slices, robustness, tail latency, throughput, memory, power, model size, preprocessing and postprocessing cost, and uncertainty across runs. Leakage-resistant splits separate locations, subjects, devices, and time where needed; confidence intervals and error taxonomies expose whether a headline score represents deployable behavior.
**Implementation, hardware, and failure modes.** High-resolution inputs improve small joints but cost memory; heatmap supervision gives spatial structure; coordinate regression is compact; flip and scale augmentation, synthetic bodies, motion data, kinematic constraints, bone-length priors, distillation, and quantization improve deployment. Multi-person top-down compute scales with detected people, while bottom-up costs more independently of count. Edge NPUs accelerate backbones; decode, association, tracking, camera transforms, and rendering can dominate CPU latency. Self-occlusion, crowding, loose clothing, unusual bodies, fast motion, motion blur, camera foreshortening, mirrored limbs, cropped people, keypoint taxonomy mismatch, biased training, and depth ambiguity create anatomically plausible but wrong poses. Engineering must include data movement, finite precision, resource contention, numerical or physical limits, error propagation, and deterministic behavior when assumptions are violated. The pipeline includes sensing, synchronization, calibration, ingestion, annotation, augmentation, training, evaluation, compilation, quantization, serving, monitoring, feedback, rollback, and dataset/model retirement. Raw data, labels, ontology versions, transforms, checkpoints, compiler artifacts, thresholds, and hardware profiles are traceable so a field failure can be reproduced.
**Evaluation, verification, and deployment.** Evaluate people- and scene-separated clips, crowd density, body diversity, clothing, camera angle, motion, occlusion, low light, temporal jitter, downstream action impact, and target-device full-pipeline latency. For 3D, coordinate alignment conventions must be reported. Pose feeds activity recognition, biomechanical metrics, avatars, gesture control, safety zones, and robot planning. Camera calibration, synchronization, floor plane, smoothing, latency, and privacy affect downstream meaning. Body and activity signals can reveal health, disability, identity, and behavior. Consent, minimization, on-device processing, retention, access, demographic performance, and non-surveillance alternatives matter. Verification combines held-out and out-of-distribution sets, synthetic stress with real validation, adversarial and corruption tests, calibration analysis, edge-case replay, hardware-in-the-loop timing, long-duration soak, human review, and shadow or canary deployment. Failures feed collection and labeling rather than being hidden by aggregate averages. The pipeline includes sensing, synchronization, calibration, ingestion, annotation, augmentation, training, evaluation, compilation, quantization, serving, monitoring, feedback, rollback, and dataset/model retirement. Raw data, labels, ontology versions, transforms, checkpoints, compiler artifacts, thresholds, and hardware profiles are traceable so a field failure can be reproduced. Evaluation reports task quality, calibration, subgroup and condition slices, robustness, tail latency, throughput, memory, power, model size, preprocessing and postprocessing cost, and uncertainty across runs. Leakage-resistant splits separate locations, subjects, devices, and time where needed; confidence intervals and error taxonomies expose whether a headline score represents deployable behavior.
| Method | Processing order | Strength | Scaling trait | Primary limitation |
|---|---|---|---|---|
| Top-down | Detect person then joints | High per-person accuracy | Cost grows with people | Detector dependency |
| Bottom-up | Detect joints then group | Efficient crowds | Image-level fixed backbone | Association errors |
| Direct regression | Predict coordinates | Compact/simple decode | Fast | Weaker spatial precision |
| Heatmap model | Predict spatial likelihood | Accurate localization | Memory-heavy output | Resolution cost |
| Temporal/3D | Use frame sequence/geometry | Stable 3D motion | Adds state/latency | Depth and calibration |
```svg
```
**Selection and practical application.** Top-down models suit accuracy with few people, bottom-up models suit crowds, regression suits compact latency, and temporal/3D systems suit motion analysis when cameras and calibration support them. Coaching, rehabilitation, gesture interfaces, motion capture, workplace ergonomics, collaborative robots, sports broadcast, and sign-language research use keypoint skeletons. Cameras, lidar, radar, IMUs, optics, illumination, clocks, mounts, compute, memory, interconnect, thermal limits, middleware, trackers, maps, planning, UI, and human escalation form one system. A faster neural network may not reduce end-to-end latency if decode, transfer, synchronization, or postprocessing dominates. A production perception claim specifies the sensor, scene distribution, label ontology, spatial and temporal resolution, operating range, latency deadline, target hardware, confidence policy, and consequence of a miss or false alarm. Dataset accuracy alone is insufficient when lighting, weather, motion, occlusion, calibration, geography, demographics, and sensor aging differ from the benchmark. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.