Open Problems

Hand-Object Interaction & Affordance

Robust 3D Hand-Object Pose and Interaction Estimation Under Upstream 2D Segmentation and Localization Noise

Effect to explainOpen
Possible candidate · 3/4 runs9 papers report this56% from 2025+

Generated automatically from the limitations stated in 9 papers (ICLR, CVPR, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current hand-object interaction and 3D hand pose estimation models rely heavily on upstream 2D bounding box crops, hand segmentation masks, or pre-extracted interaction regions. In real-world egocentric scenes, upstream 2D segmenters and detectors degrade severely due to object occlusions, motion blur, and cluttered backgrounds, causing downstream 3D estimation to fail or hallucinate. Because models are trained and benchmarked assuming near-ideal 2D crops or masks, existing pipelines cannot operate reliably in autonomous, in-the-wild video streams.

Why it matters

End-to-end hand-object pose and affordance estimation directly from uncurated egocentric video streams without requiring clean manual prompts, high-accuracy 2D segmentation masks, or heuristic threshold tuning.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark degradation by evaluating state-of-the-art 3D hand/HOI estimators under controlled perturbations of upstream 2D bounding boxes and segmentation masks (dilation, erosion, boundary noise, and synthetic occlusion) on datasets like VISOR and HO-3D to quantify error propagation.

  2. 2

    Train end-to-end full-image HOI architectures that integrate soft, multi-scale hand attention mechanisms instead of hard bounding box crops or binary masks, measuring 3D joint error under noisy detector proposals.

  3. 3

    Design occlusion-aware feature aggregation that conditions joint estimation on temporal optical flow and kinematic priors rather than local mask resamplers, measuring pose accuracy during severe object occlusion phases (e.g., manipulation, folding).

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If upstream foundation models (e.g., future iterations of SAM or open-world video object segmenters) become sufficiently robust to cluttered egocentric hands and severe object occlusions that upstream segmentation noise ceases to be a failure mode in practice.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 9 papers

Nearest existing work

Related open problems

Hand-Object Interaction & Affordance

Barrier to removeOpen

Robust Hand-Object Interaction Learning Beyond Cascaded Parametric Pose Estimators

Downstream hand-object interaction (HOI) and affordance models are universally bottlenecked by a strict reliance on upstream off-the-shelf 3D hand pose estimators (e.g., FrankMocap, HaMeR, MANO parameterizations) and bounding-box detectors. When these upstream tools encounter severe object occlusions, fast manipulation dynamics, or egocentric perspective distortion, their estimation errors cascade directly into downstream HOI training, causing catastrophic mis-projections and synthetic contact artifacts. Because downstream methods require clean parametric 3D meshes as preconditions, massive repositories of in-the-wild manipulation videos cannot be utilized without fragile filtering heuristics that discard over half the data.

Possible candidate · 2/5 runs42 papers report this45% from 2025+

Hand-Object Interaction & Affordance

Scope to testPartly addressed

Benchmarking Hand-Object Interaction Models Across Diverse Hand Morphologies and Embodiments

Existing hand representation and hand-object interaction models are predominantly evaluated on small demographic cohorts or fixed kinematic setups (e.g., standard anthropomorphic models like MANO or fixed robotic hands like ShadowHand). Because existing pipelines implicitly bind kinematics, palm-to-finger ratios, or subject identities into their representations, their performance on unseen human hand proportions and non-standard robotic end-effectors remains untested. As a result, it is unknown whether current interaction and affordance estimators degrade gracefully or fail catastrophically when applied outside their narrow training morphologies.

Possible candidate · 2/5 runs7 papers report this29% from 2025+

Hand-Object Interaction & Affordance

Barrier to removeOpen

Fine-Grained 3D Hand Affordance Grounding and Generation for Small-Part Articulated Interactions

Current affordance learning pipelines rely on self-training loops, video extraction, or upstream generative HOI models that fail when contact targets are small functional components such as bottle caps, zippers, or rotary knobs. Because generated hand poses easily drift to adjacent object surfaces, models exhibit catastrophic failure on precision affordances (e.g., 0% accuracy on pull or sub-50% on twist), propagating noisy pseudo-labels across unobserved geometries. Downstream manipulation systems are consequently blocked from executing precision tasks like uncapping, pulling sliders, or fine bimanual manipulation.

Possible candidate · 3/5 runs4 papers report this75% from 2025+

Hand-Object Interaction & Affordance

Barrier to removeOpen

Hand-Object Affordance Estimation Beyond Clean Pre-Scanned Scenes and Predefined 3D Grids

Current methods for 3D hand-object interaction and affordance estimation depend strictly on clean geometric preconditions: high-fidelity 3D scans, pre-detected object instances, static overhead views, or tightly predefined 3D bounding grids. In practical manipulation and augmented reality scenarios, environments are unsegmented, viewpoints are dynamic, and point clouds are noisy and incomplete. Relying on manually tuned grid volumes and pre-extracted meshes prevents existing models from operating directly on raw, uncurated sensor streams.

Strong candidate · 5/5 runs3 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.