Open Problems

3D Point Clouds & LiDAR Perception

Benchmark and Training Protocols for 3D Perception from Unpaired or Calibration-Free 2D-3D Data

Barrier to removePartly addressed
Possible candidate · 2/5 runs11 papers report this17% from 2025+

Generated automatically from the limitations stated in 11 papers (CVPR, ICCV, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Nearly every current 2D-3D fusion method assumes registered multi-view imagery, known intrinsics/extrinsics, and per-frame paired data — assumptions that hold only in curated lab captures. The consequence is that point clouds collected standalone (LiDAR scans, crowdsourced reconstructions, legacy datasets) cannot benefit from image features at all, and methods cannot be deployed where a fixed calibrated rig with synchronized cameras is impractical. Progress is blocked because no established protocol or dataset exists for learning 2D-3D correspondences from unpaired, uncalibrated, or single-view inputs.

Why it matters

Methods that work on raw point clouds and arbitrary image configurations, enabling deployment on LiDAR vehicles, monocular robots, and uncalibrated consumer captures without re-engineering per rig.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Build an evaluation suite that degrades pairing: start from paired datasets (ScanNet-style), then systematically remove calibration, drop views to single-image, and jitter extrinsics; measure AP/recall curves as a function of pairing quality to quantify how much of current methods' performance is attributable to registration rather than to learned features.

  2. 2

    Develop and test calibration-free distillation: train a point-cloud-only student network to match the features of a paired-image teacher, using self-supervised alignment losses; measure how close student performance on raw point clouds gets to the teacher's paired-input performance on standard benchmarks.

  3. 3

    Explore test-time calibration estimation — predicting intrinsics/extrinsics from the data itself as a preprocessing module — and measure end-task recovery when its estimates substitute for ground-truth calibration.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Distilled point-cloud students may plateau far below paired-input performance because image features genuinely encode geometry absent from sparse point clouds, meaning the barrier is information-theoretic rather than methodological — in which case the degradation benchmark is the only durable contribution.

Sub-problems

  • Hand-Object Interaction and Affordance Learning Without Calibrated Multi-View Camera Rigs

    Current 3D hand-object interaction and affordance learning methods require synchronized, calibrated multi-view camera arrays or fixed multi-camera rigs during training and inference for geometric triangulation. This dependence restricts research to lab capture domes and prevents these models from operating on standard egocentric head-mounted displays or monocular consumer video. Consequently, models cannot generalize to unconstrained in-the-wild environments where synchronous multi-view calibration data does not exist.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 11 papers

Nearest existing work

Related open problems

3D Point Clouds & LiDAR Perception

Scope to testOpen

An Outdoor-Scale Benchmark for 3D Scene Understanding Methods Currently Validated Only on Indoor ScanNet-Derived Data

Nearly the entire 3D perception, scene reconstruction, grounding, and question-answering stack is trained and evaluated on ScanNet-family indoor scans, so claims of scene understanding hold only within bounded, static, room-scale environments with known camera poses. As a result, no one can tell whether reported architectural advances—reasoning modules, open-vocabulary grounding, generalization arguments—transfer to outdoor, unbounded, or unsegmented scenes at all, or whether they simply exploit ScanNet's statistics. Progress reports that look cumulative within this benchmark family may be incommensurable with outdoor work that cannot reuse the same evaluation. Any paper today claiming 3D scene understanding is doing so on a single environmental regime.

Possible candidate · 3/5 runs39 papers report this30% from 2025+

3D Point Clouds & LiDAR Perception

Scope to testOpen

Benchmarking and Adapting Indoor-Trained 3D Perception and Generation Models to Outdoor and LiDAR Point Cloud Scenes

Nearly every 3D foundation-ish model — for detection, segmentation, registration, scene generation, and novel view synthesis — is trained and validated exclusively on indoor RGB-D or synthetic benchmarks (ScanNet, SUN RGB-D, 3D-FRONT, Habitat simulators), with explicit author acknowledgments that outdoor and LiDAR data is untested. This means the field's core 3D representations are calibrated to room-scale geometry (bounded depth, dense views, flat priors like indoor depth predictors and category vocabularies), and nobody can currently deploy them on autonomous-driving or aerial scans without unquantified degradation. The consequence is a silent incompatibility: outdoor researchers must either rebuild models from scratch or trust unmeasured transfer.

Possible candidate · 3/5 runs20 papers report this53% from 2025+

3D Point Clouds & LiDAR Perception

Barrier to removeOpen

Robust Multimodal 3D Perception Under Sensor Extrinsic and Intrinsic Miscalibration

State-of-the-art multimodal 3D perception architectures strictly require accurate extrinsic and intrinsic calibration between LiDAR and camera frames to construct point-to-pixel projections, frustum cost volumes, and cross-attention neighbourhoods. In real-world deployments, mechanical vibration, thermal expansion, and mounting shifts cause calibration parameters to drift, leading to severe geometric misalignment. Because current multimodal models hardcode these spatial transformations into their feature lifting layers, minor calibration errors cause catastrophic performance degradation or complete pipeline failure.

Possible candidate · 2/5 runs13 papers report this38% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.