Open Problems

Hand-Object Interaction & Affordance

Hand-Object Affordance Estimation Beyond Clean Pre-Scanned Scenes and Predefined 3D Grids

Barrier to removeOpen
Strong candidate · 5/5 runs3 papers report this33% from 2025+

Generated automatically from the limitations stated in 3 papers (CVPR, NeurIPS, ICCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current methods for 3D hand-object interaction and affordance estimation depend strictly on clean geometric preconditions: high-fidelity 3D scans, pre-detected object instances, static overhead views, or tightly predefined 3D bounding grids. In practical manipulation and augmented reality scenarios, environments are unsegmented, viewpoints are dynamic, and point clouds are noisy and incomplete. Relying on manually tuned grid volumes and pre-extracted meshes prevents existing models from operating directly on raw, uncurated sensor streams.

Why it matters

Enables robotic systems and wearable AR devices to predict contact affordances and interaction poses directly from raw sensor observations in unmapped, uncurated environments without requiring pre-scanned 3D assets or pre-segmented bounding boxes.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark existing affordance models under degraded, multi-view, and unsegmented input geometry (e.g., raw depth video without pre-cropped bounding boxes or pre-detected masks) to quantify performance degradation across varying noise and missing-surface levels.

  2. 2

    Develop an anchor-free affordance estimation framework that predicts interaction regions directly on continuous, sensor-frame coordinate spaces rather than discrete, pre-sized voxel grids.

  3. 3

    Integrate joint object localization and contact affordance reasoning into a single-stage model that avoids the requirement for pre-detected object and container boundaries.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If end-to-end 2D-to-3D vision-language-action models bypass explicit 3D affordance representation entirely, rendering standalone 3D affordance estimation pipelines obsolete before geometry-free methods mature.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Hand-Object Interaction & Affordance

Barrier to removeOpen

Robust Hand-Object Interaction Learning Beyond Cascaded Parametric Pose Estimators

Downstream hand-object interaction (HOI) and affordance models are universally bottlenecked by a strict reliance on upstream off-the-shelf 3D hand pose estimators (e.g., FrankMocap, HaMeR, MANO parameterizations) and bounding-box detectors. When these upstream tools encounter severe object occlusions, fast manipulation dynamics, or egocentric perspective distortion, their estimation errors cascade directly into downstream HOI training, causing catastrophic mis-projections and synthetic contact artifacts. Because downstream methods require clean parametric 3D meshes as preconditions, massive repositories of in-the-wild manipulation videos cannot be utilized without fragile filtering heuristics that discard over half the data.

Possible candidate · 2/5 runs42 papers report this45% from 2025+

Hand-Object Interaction & Affordance

Effect to explainOpen

Robust 3D Hand-Object Pose and Interaction Estimation Under Upstream 2D Segmentation and Localization Noise

Current hand-object interaction and 3D hand pose estimation models rely heavily on upstream 2D bounding box crops, hand segmentation masks, or pre-extracted interaction regions. In real-world egocentric scenes, upstream 2D segmenters and detectors degrade severely due to object occlusions, motion blur, and cluttered backgrounds, causing downstream 3D estimation to fail or hallucinate. Because models are trained and benchmarked assuming near-ideal 2D crops or masks, existing pipelines cannot operate reliably in autonomous, in-the-wild video streams.

Possible candidate · 3/4 runs9 papers report this56% from 2025+

Hand-Object Interaction & Affordance

Scope to testPartly addressed

Benchmarking Hand-Object Interaction Models Across Diverse Hand Morphologies and Embodiments

Existing hand representation and hand-object interaction models are predominantly evaluated on small demographic cohorts or fixed kinematic setups (e.g., standard anthropomorphic models like MANO or fixed robotic hands like ShadowHand). Because existing pipelines implicitly bind kinematics, palm-to-finger ratios, or subject identities into their representations, their performance on unseen human hand proportions and non-standard robotic end-effectors remains untested. As a result, it is unknown whether current interaction and affordance estimators degrade gracefully or fail catastrophically when applied outside their narrow training morphologies.

Possible candidate · 2/5 runs7 papers report this29% from 2025+

Hand-Object Interaction & Affordance

Barrier to removeOpen

Fine-Grained 3D Hand Affordance Grounding and Generation for Small-Part Articulated Interactions

Current affordance learning pipelines rely on self-training loops, video extraction, or upstream generative HOI models that fail when contact targets are small functional components such as bottle caps, zippers, or rotary knobs. Because generated hand poses easily drift to adjacent object surfaces, models exhibit catastrophic failure on precision affordances (e.g., 0% accuracy on pull or sub-50% on twist), propagating noisy pseudo-labels across unobserved geometries. Downstream manipulation systems are consequently blocked from executing precision tasks like uncapping, pulling sliders, or fine bimanual manipulation.

Possible candidate · 3/5 runs4 papers report this75% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.