Open Problems

3D Point Clouds & LiDAR Perception

Robust Multimodal 3D Perception Under Sensor Extrinsic and Intrinsic Miscalibration

Barrier to removeOpen
Possible candidate · 2/5 runs13 papers report this38% from 2025+

Generated automatically from the limitations stated in 13 papers (CVPR, ICCV, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

State-of-the-art multimodal 3D perception architectures strictly require accurate extrinsic and intrinsic calibration between LiDAR and camera frames to construct point-to-pixel projections, frustum cost volumes, and cross-attention neighbourhoods. In real-world deployments, mechanical vibration, thermal expansion, and mounting shifts cause calibration parameters to drift, leading to severe geometric misalignment. Because current multimodal models hardcode these spatial transformations into their feature lifting layers, minor calibration errors cause catastrophic performance degradation or complete pipeline failure.

Why it matters

Enables robust deployment of multimodal 3D perception systems on commercial vehicle fleets and robotics platforms subject to continuous sensor vibration, mechanical drift, and imperfect factory calibration without requiring frequent offline recalibration.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark existing multimodal 3D detectors (e.g., BEVFusion, PointPainting, TransFusion) under synthetic and real-world extrinsic and intrinsic perturbation regimes (rotation/translation jitter and focal drift), quantifying the exact degradation curves across tasks.

  2. 2

    Develop a deformable or cross-attention-based fusion module that learns soft correspondences between unprojected LiDAR coordinates and multi-view image features, removing the hard geometric projection precondition.

  3. 3

    Design an online, self-supervised calibration refinement layer integrated directly into the 3D perception backbone that dynamically updates relative sensor extrinsics using multimodal feature consistency.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

If end-to-end online autocalibration or visual-geometric foundation models advance rapidly enough to guarantee sub-pixel real-time sensor alignment as a trivial pre-processing step, rendering calibration-invariant perception architectures redundant.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 13 papers

Nearest existing work

Related open problems

3D Point Clouds & LiDAR Perception

Scope to testOpen

An Outdoor-Scale Benchmark for 3D Scene Understanding Methods Currently Validated Only on Indoor ScanNet-Derived Data

Nearly the entire 3D perception, scene reconstruction, grounding, and question-answering stack is trained and evaluated on ScanNet-family indoor scans, so claims of scene understanding hold only within bounded, static, room-scale environments with known camera poses. As a result, no one can tell whether reported architectural advances—reasoning modules, open-vocabulary grounding, generalization arguments—transfer to outdoor, unbounded, or unsegmented scenes at all, or whether they simply exploit ScanNet's statistics. Progress reports that look cumulative within this benchmark family may be incommensurable with outdoor work that cannot reuse the same evaluation. Any paper today claiming 3D scene understanding is doing so on a single environmental regime.

Possible candidate · 3/5 runs39 papers report this30% from 2025+

3D Point Clouds & LiDAR Perception

Scope to testOpen

Benchmarking and Adapting Indoor-Trained 3D Perception and Generation Models to Outdoor and LiDAR Point Cloud Scenes

Nearly every 3D foundation-ish model — for detection, segmentation, registration, scene generation, and novel view synthesis — is trained and validated exclusively on indoor RGB-D or synthetic benchmarks (ScanNet, SUN RGB-D, 3D-FRONT, Habitat simulators), with explicit author acknowledgments that outdoor and LiDAR data is untested. This means the field's core 3D representations are calibrated to room-scale geometry (bounded depth, dense views, flat priors like indoor depth predictors and category vocabularies), and nobody can currently deploy them on autonomous-driving or aerial scans without unquantified degradation. The consequence is a silent incompatibility: outdoor researchers must either rebuild models from scratch or trust unmeasured transfer.

Possible candidate · 3/5 runs20 papers report this53% from 2025+

3D Point Clouds & LiDAR Perception

Barrier to removePartly addressed

Benchmark and Training Protocols for 3D Perception from Unpaired or Calibration-Free 2D-3D Data

Nearly every current 2D-3D fusion method assumes registered multi-view imagery, known intrinsics/extrinsics, and per-frame paired data — assumptions that hold only in curated lab captures. The consequence is that point clouds collected standalone (LiDAR scans, crowdsourced reconstructions, legacy datasets) cannot benefit from image features at all, and methods cannot be deployed where a fixed calibrated rig with synchronized cameras is impractical. Progress is blocked because no established protocol or dataset exists for learning 2D-3D correspondences from unpaired, uncalibrated, or single-view inputs.

Possible candidate · 2/5 runs11 papers report this17% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.