Open Problems

Hand-Object Interaction & Affordance

Fine-Grained 3D Hand Affordance Grounding and Generation for Small-Part Articulated Interactions

Barrier to removeOpen
Possible candidate · 3/5 runs4 papers report this75% from 2025+

Generated automatically from the limitations stated in 4 papers (CVPR, ICLR, ICCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Current affordance learning pipelines rely on self-training loops, video extraction, or upstream generative HOI models that fail when contact targets are small functional components such as bottle caps, zippers, or rotary knobs. Because generated hand poses easily drift to adjacent object surfaces, models exhibit catastrophic failure on precision affordances (e.g., 0% accuracy on pull or sub-50% on twist), propagating noisy pseudo-labels across unobserved geometries. Downstream manipulation systems are consequently blocked from executing precision tasks like uncapping, pulling sliders, or fine bimanual manipulation.

Why it matters

Enables robotic and generative systems to accurately localize and synthesize viable hand interactions on millimeter-to-centimeter scale functional object parts, unlocking precision manipulation tasks such as unscrewing caps, unzipping bags, and operating small switches.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Contact-constrained local refinement: Develop a local geometry-aware refinement module that conditions hand pose and affordance heatmaps on high-resolution local surface curvature and keypoint signifiers rather than global object representations, evaluated by contact intersection volume and part-level affordance precision on AffordPose and SceneFun3D benchmarks.

  2. 2

    Signifier-guided pseudo-label filtering: Build a geometric consistency verifier that filters noisy pseudo-labels in self-training loops using functional affordance priors (e.g., surface normals, clearance margins), measured by label error rate and generalization accuracy on out-of-distribution small-part test shapes.

  3. 3

    Multi-view fine-grained interaction bootstrapping: Synthesize precision contact regions by projecting fine-scale multi-view contact trajectories into 3D signed distance fields, measured by affordance mIoU and downstream grasp success for precision affordance classes (twist, pull, pinch).

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Sensor noise and low point cloud resolution in real-world 3D scans might fundamentally obscure millimeter-scale parts like zipper tabs, rendering purely geometric refinement insufficient without specialized macro-sensing.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Hand-Object Interaction & Affordance

Barrier to removeOpen

Robust Hand-Object Interaction Learning Beyond Cascaded Parametric Pose Estimators

Downstream hand-object interaction (HOI) and affordance models are universally bottlenecked by a strict reliance on upstream off-the-shelf 3D hand pose estimators (e.g., FrankMocap, HaMeR, MANO parameterizations) and bounding-box detectors. When these upstream tools encounter severe object occlusions, fast manipulation dynamics, or egocentric perspective distortion, their estimation errors cascade directly into downstream HOI training, causing catastrophic mis-projections and synthetic contact artifacts. Because downstream methods require clean parametric 3D meshes as preconditions, massive repositories of in-the-wild manipulation videos cannot be utilized without fragile filtering heuristics that discard over half the data.

Possible candidate · 2/5 runs42 papers report this45% from 2025+

Hand-Object Interaction & Affordance

Effect to explainOpen

Robust 3D Hand-Object Pose and Interaction Estimation Under Upstream 2D Segmentation and Localization Noise

Current hand-object interaction and 3D hand pose estimation models rely heavily on upstream 2D bounding box crops, hand segmentation masks, or pre-extracted interaction regions. In real-world egocentric scenes, upstream 2D segmenters and detectors degrade severely due to object occlusions, motion blur, and cluttered backgrounds, causing downstream 3D estimation to fail or hallucinate. Because models are trained and benchmarked assuming near-ideal 2D crops or masks, existing pipelines cannot operate reliably in autonomous, in-the-wild video streams.

Possible candidate · 3/4 runs9 papers report this56% from 2025+

Hand-Object Interaction & Affordance

Scope to testPartly addressed

Benchmarking Hand-Object Interaction Models Across Diverse Hand Morphologies and Embodiments

Existing hand representation and hand-object interaction models are predominantly evaluated on small demographic cohorts or fixed kinematic setups (e.g., standard anthropomorphic models like MANO or fixed robotic hands like ShadowHand). Because existing pipelines implicitly bind kinematics, palm-to-finger ratios, or subject identities into their representations, their performance on unseen human hand proportions and non-standard robotic end-effectors remains untested. As a result, it is unknown whether current interaction and affordance estimators degrade gracefully or fail catastrophically when applied outside their narrow training morphologies.

Possible candidate · 2/5 runs7 papers report this29% from 2025+

Hand-Object Interaction & Affordance

Barrier to removeOpen

Hand-Object Affordance Estimation Beyond Clean Pre-Scanned Scenes and Predefined 3D Grids

Current methods for 3D hand-object interaction and affordance estimation depend strictly on clean geometric preconditions: high-fidelity 3D scans, pre-detected object instances, static overhead views, or tightly predefined 3D bounding grids. In practical manipulation and augmented reality scenarios, environments are unsegmented, viewpoints are dynamic, and point clouds are noisy and incomplete. Relying on manually tuned grid volumes and pre-extracted meshes prevents existing models from operating directly on raw, uncurated sensor streams.

Strong candidate · 5/5 runs3 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.