Open Problems

Camera Pose, SLAM & Structure-from-Motion

Reducing Reliance on Pre-Built 3D Supervision for Training 3D-Aware Perception and Reconstruction Models

Barrier to removePartly addressed
Possible candidate · 3/4 runs24 papers report this50% from 2025+

Generated automatically from the limitations stated in 24 papers (CVPR, ICCV, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Nearly every system that connects 2D images to 3D representations — whether for localization, reconstruction, or generation — is trained against 3D artifacts that must be manufactured in advance: dense depth, camera poses, meshes, point clouds, scene graphs, or splat reconstructions from dense multi-view rigs. This makes training data a bottleneck that caps scalability to ordinary single-view RGB collections, blocks application to domains without depth sensors or calibrated rigs, and drives enormous compute costs for pseudo-ground-truth generation. Until models can learn the 2D-to-3D link without manufactured 3D supervision, the field's coverage is limited to the small slice of the visual world that has been 3D-digitized.

Why it matters

Training 3D-aware models directly on uncalibrated single-view image and video collections at scale, removing dependence on depth sensors, calibration rigs, and offline 3D reconstruction of training scenes.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Systematic ablation of 3D supervision across a fixed backbone: take one representative 2D-to-3D architecture and retrain it with progressively weaker supervision — full GT 3D, pseudo-GT from off-the-shelf depth/pose estimators, self-supervised multi-view consistency losses only, and single-view-only — measuring task accuracy, generalization to unseen domains, and total training compute. This quantifies which parts of the 3D stack are actually load-bearing and which are trainable artifacts. Measured: accuracy-vs-supervision curves and data-sourcing cost.

  2. 2

    Pseudo-3D label generation at web scale: build a pipeline that converts uncalibrated internet video into weak 3D supervision (relative poses, sparse depth, covisibility) using self-supervised pose estimators and monocular depth, with confidence filtering, and measure the tradeoff between pseudo-label volume/quality and downstream performance. Measured: downstream task accuracy as a function of pseudo-label noise and scale.

  3. 3

    Supervision-free 2D-3D consistency objectives: train with losses that enforce geometric consistency (epipolar constraints, cross-view photometric agreement, cycle-consistency between 2D observations and predicted 3D structure) without any explicit 3D labels, and benchmark against supervised counterparts on standard localization and reconstruction metrics. Measured: gap to fully-supervised baselines.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The ablations may show that high-quality 3D ground truth is genuinely load-bearing, with consistency-only training collapsing to degenerate solutions, meaning the barrier is fundamental rather than a data-manufacturing problem.

Sub-problems

  • Learning Feed-Forward 3D Gaussian Splatting from Unposed 2D Image and Video Collections

    Current feed-forward 3D Gaussian Splatting and novel view synthesis architectures require calibrated multi-view captures, RGB-D inputs, or pre-optimized 3D target ground truths during training. This creates a hard barrier against training on internet-scale passive 2D video and uncalibrated image collections. Consequently, models remain confined to small synthetic or specialized multi-camera lab datasets, suffering severe generalization collapse when applied to arbitrary single-view or in-the-wild camera trajectories.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 24 papers

Nearest existing work

Related open problems

Camera Pose, SLAM & Structure-from-Motion

Barrier to removeOpen

Robust Pose and 3D Structure Estimation Under Imperfect and Open-World Instance Masks

Current object-centric camera pose estimation, SLAM, and structure-from-motion pipelines treat 2D or 3D instance segmentation as a frozen upstream precondition (predominantly off-the-shelf Mask R-CNN or Mask3D). When masks contain boundary leakage, missed detections, or category misclassifications, geometric estimation degrades sharply because segmentation noise directly injects outlier correspondences that standard robust estimators cannot fully filter out. Consequently, existing methods cannot operate reliably outside closed object benchmarks with clean instance masks.

Strong candidate · 5/5 runs6 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.