Robust 3D Pose Lifting Under Empirical Upstream 2D Detector Errors

The problem

Monocular 3D human pose estimators depend fundamentally on upstream 2D keypoint detectors, suffering severe accuracy degradation (e.g., MPJPE increases of 40% to over 100%) when moving from ground-truth 2D annotations to actual detector outputs. Current lifting architectures are predominantly trained on clean annotations or evaluated against simplistic Gaussian perturbation, leaving them brittle to real detector artifacts such as keypoint swaps, missing joints from occlusion, and redundant bounding boxes. Consequently, isolated benchmark gains on clean 3D datasets fail to translate into robust end-to-end motion capture in unconstrained real-world environments.

Why it matters

Enables 2D-to-3D lifting systems to deploy reliably on uncurated, in-the-wild video feeds without manual annotation or catastrophic failure from upstream 2D detection noise.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

1. 1

   Benchmark Realistic Detector Failure Modes: Profile the empirical error distributions of standard 2D detectors (e.g., ViTPose, HRNet, MediaPipe, CPN) across Human3.6M and MPI-INF-3DHP, and systematically evaluate the degradation of current lifting architectures against these realistic error profiles rather than synthetic Gaussian noise.

   Check for prior work
2. 2

   Detector-Aware Data Augmentation: Train existing 2D-to-3D lifters using an empirical noise model parameterized by detector uncertainty and topological keypoint dropouts, measuring 3D MPJPE recovery on uncurated video detections.

   Check for prior work
3. 3

   Joint Uncertainty-Aware Keypoint Imputation: Develop a lightweight keypoint refinement module that consumes 2D detector confidence scores to mask and impute unreliable joints prior to 3D lifting, measuring end-to-end MPJPE across multiple detector backbones.

   Check for prior work

Why it might fail

End-to-end direct image-to-3D mesh foundation models may rapidly mature, completely bypassing the two-stage 2D-to-3D lifting pipeline and rendering upstream 2D detector mitigation obsolete.

Evidence

Each paper's own statement of the limitation, verbatim.

• VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention NetworkCVPR 2026

  Significant accuracy degradation when upstream 2D pose detection quality degrades (e.g., MPJPE increases from 17.2 mm with ground truth to 31.2 mm with SimpleNet and 33.2 mm with MediaPipe).

• VOccl3D: A Video Benchmark Dataset for 3D Human Pose and Shape Estimation under real OcclusionsICCV 2025

  Detector failures dominate end-to-end performance (MPJPE jumps from ~71 with GT boxes to ~115 with predicted detections), so dataset gains in pose estimation alone do not yield a robust full system

• PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor SpaceNeurIPS 2025

  Depends on a preceding 2D pose estimator whose quality affects input; robustness is improved but degraded 2D poses still reduce accuracy (only Gaussian-noise robustness curves reported)

• ExtPose: Robust and Coherent Pose Estimation by Extending ViTsICML 2025

  Depends on an off-the-shelf 2D pose detector (ViTPose); its errors propagate, and with GT 2D poses PA-MPJPE drops to 3.7mm on FreiHAND, so performance is bounded by 2D detection quality

• Dynamic Camera Poses and Where to Find ThemCVPR 2025

  Filtering precision is 0.78, so roughly 22% of collected videos may still be unsuitable for pose estimation

• DynPose: Largely Improving the Efficiency of Human Pose Estimation by a Simple Dynamic FrameworkCVPR 2025

  'Extremely hard' samples, largely caused by redundant detector boxes, are discarded rather than fixed — accuracy on genuinely ambiguous poses is not improved

• SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose EstimationCVPR 2024

  Quality is bounded by the off-the-shelf 2D detector: using ground-truth 2D poses instead drops MPJPE from 24.5 to 19.9mm, and better COCO detectors (HRNet 76.3 AP vs Keypoint R-CNN 66.1 AP) directly improve results

• KITRO: Refining Human Mesh by 2D Clues and Kinematic-tree RotationCVPR 2024

  Depends on accuracy of input 2D keypoints; noisy or detected (rather than ground-truth) 2D poses degrade refinement

Nearest existing work

• A Simple yet Effective Baseline for 3D Human Pose EstimationICCV 2017
• Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose EstimationCVPR 2022
• Deep Kinematics Analysis for Monocular 3D Human Pose EstimationCVPR 2020
• On Boosting Single-Frame 3D Human Pose Estimation via Monocular VideosICCV 2019
• Towards Alleviating the Modeling Ambiguity of Unsupervised Monocular 3D Human Pose EstimationICCV 2021
• Sim2real transfer learning for 3D human pose estimation: motion to the rescueNeurIPS 2019
• PoseSyn: Synthesizing Diverse 3D Pose Data from In-the-Wild 2D DataICCV 2025
• Towards Robust and Smooth 3D Multi-Person Pose Estimation from Monocular Videos in the WildICCV 2023
• PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose EstimationCVPR 2023
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and BenchmarksCVPR 2021
• Unsupervised 3D Pose Estimation With Geometric Self-SupervisionCVPR 2019
• In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D RepresentationsCVPR 2019
• SynSP: Synergy of Smoothness and Precision in Pose Sequences RefinementCVPR 2024
• GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular VideoICCV 2023
• PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor SpaceNeurIPS 2025