Robust Hand-Object Interaction Learning Beyond Cascaded Parametric Pose Estimators

Requires known or pre-computed MANO hand pose parameters and 6-DoF object poses for canonical space mapping and initialization.

Cannot train directly on datasets lacking MANO hand parameterizations or containing noisy hand annotations (e.g., AffordPose).

Requires HaMeR-based 3D hand reconstruction, pose estimation, and DensePose as inputs, so errors in these upstream parsers propagate; the paper notes existing parsing models fail on tiny fragmented finger areas

No frame-level handshape annotations, so static-frame selection relies on a crude minimum-displacement heuristic that may miss the canonical frame

3D hand detection (HaMeR) errors propagate into the dataset, causing inaccurate hand motions and desynchronized predictions when hands are less visible or motion is ambiguous

Depends on an external hand detector/tracker; method evaluated only on DexYCB and HOT3D benchmarks

Object geometry must be available as a point cloud at encoding/decoding time, and hand poses depend on MANO parameterization, so arbitrary objects/hand representations are out of scope

Assumes rigid objects and SMPL-X humans with coarse hand representations, causing minor inaccuracies in hand-object interactions, particularly rotations

The dataset excludes hand/finger motions, so generated hand details are low-fidelity

Conditioning requires 2D keypoints; the paper notes hand masks and keypoint images can be ambiguous under heavy finger occlusion, and NVS depends on an off-the-shelf 3D pose estimator with assumed camera intrinsics

Hand segmentation quality is substantially below supervised upper bound (61.01 vs 90.87 hand-AP on VISOR) due to noisy HOI detection prompts and SAM2 struggles in cluttered egocentric backgrounds

Restricted to static hand knowledge; hand dynamics from video are not addressed

Requires an external hand detector and handedness classifier at test time, and handles only one hand per crop (two-hand scenes processed independently)

Hand-object interactions not modeled despite datasets containing object pose information

Depends on hand pose availability; poses estimated from visual inputs suffer from out-of-view cases and noise

Hand poses are set to an average pose because the BEHAVE dataset lacks hand annotations, degrading interaction realism

Requires predicted hand pose (MANO parameters) and camera view from off-the-shelf estimators; uses ground-truth hand poses in ObMan evaluation, so errors in hand pose estimation are not fully assessed

Performance degrades with noisy or low-detail hand masks; more detailed segmentation masks produce finer-grained outputs

Inaccurate FrankMocap hand pose predictions cause mis-projection so sampled points do not cover the entire object, excluding parts of the object from supervision

Performance degrades when hand parts are severely occluded because the resampler cannot sample informative points near occluded joints

Focused on hands without heavy occlusions; hand-object interaction scenarios and unsynchronized multi-view inputs are excluded

Requires datasets with 3D hand and object mesh ground truth for supervision; performance on real data relies on filtered subsets (e.g., FHB excludes frames with hand-object distance >10 mm)

Relies on off-the-shelf hand pose prediction (FrankMocap) and masks; while robust to doubled hand-pose error, severe initial failures require a separate trajectory-optimization correction step

Does not fully exploit estimated 3D mesh information, which could help fine-grained inter-hand understanding

Requires a per-object optimization (200 + 1000 Adam iterations) to recover the hand pose from each sampled ContactGen

Relies on preceding hand pose estimation and category-level object pose estimation; GT-pose results (CD 2.30 vs 7.69 for mug) show large sensitivity to pose input quality

Labels come from off-the-shelf hand-object detection, pose, and egomotion estimators, so label noise from those tools propagates into training

Pseudo-ground-truth data collection depends entirely on the accuracy and error propagation of off-the-shelf active hand-object detectors and skin/fingertip segmentation.

Ground-truth hand masks do not distinguish left from right hands, so the model cannot make separate left/right hand predictions

Removing either spatial or temporal filtering constraints degrades hand pose accuracy below the supervised baseline, so the pseudo-label pipeline is fragile to filter design

Performance depends on the newly collected HIU-Data (33k manually annotated images); without it, evaluation of mask/2D pose claims against other methods is limited since no prior dataset has high-quality hand masks

Relies on a pre-cropped, normalized hand region (96x96) and assumes a fixed palm/finger joint partition, which may not transfer to hands in unconstrained scenes or other skeletal topologies

Requires hand segmentation (via a separate hourglass network on NYU/ICVL) and PCA pose prior fitting

Performance is sensitive to the accuracy of hand segmentation and configuration classification, which remains a non-trivial task.