Semantic & Instance Segmentation
Benchmarking and Developing Segmentation Without Dense Initial Annotation Preconditions
Generated automatically from the limitations stated in 3 papers (ICML, NeurIPS, ICCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Current semantic and instance segmentation methods rely on strict supervision preconditions, such as dense pixel-wise ground truth or accurate initial bounding-box-derived masks. When applied to domains where only weak annotations (such as image-level tags or sparse points) are available from scratch, these methods cannot be initialized or trained. Consequently, segmentation models cannot be deployed in annotation-scarce domains without first undertaking costly manual pixel-level or box-level labeling campaigns.
Why it matters
Enables training segmentation models directly from low-cost weak labels without requiring a dense annotation bootstrap phase. This drastically reduces the data curation cost required to train visual segmentation models in specialized application domains.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Benchmark existing segmentation architectures across a continuous spectrum of initial supervision budgets (image tags, point clicks, coarse bounding boxes, and fully dense masks) to quantify the exact performance drop-off when dense initial masks are omitted. Measure mean Intersection-over-Union (mIoU) and boundary accuracy under identical computational budgets.
- 2
Develop a self-training bootstrap mechanism that initializes pseudo-masks purely from image-level class activations or foundation model feature correspondences without requiring t=0 pixel ground truth, measuring label noise robustness and final mask mIoU on standard benchmarks (e.g., PASCAL VOC, MS COCO).
- 3
Formulate a consistency regularization framework that directly optimizes segmentation objectives using sparse or box-free supervision signals, measuring relative performance retention against fully supervised upper bounds.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
Foundation models like SAM with zero-shot prompting may commoditize mask generation from weak prompts, shifting the challenge from algorithmic formulation to prompt engineering. Alternatively, pseudo-mask noise at early iterations may consistently cause optimization collapse without at least a small set of dense anchors.
Evidence
Each paper's own statement of the limitation, verbatim.
- Learning 1-Bit Tiny Object Detector with Discriminative Feature RefinementICML 2024
Requires real-valued ground-truth foreground masks derived from boxes at training time, so the approach depends on accurate box annotations and does not address weakly labeled settings
- Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNsNeurIPS 2024
Requires fully annotated datasets; not applicable to weakly- or unsupervised settings
- Class-incremental Continual Learning for Instance Segmentation with Image-level Weak SupervisionICCV 2023
Requires fully pixel-wise annotations at the initial step (t=0), so the setting is not fully weakly supervised from scratch
Nearest existing work
- Semantic-aware SAM for Point-Prompted Instance SegmentationCVPR 2024
- BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance SegmentationCVPR 2021
- SOS: Segment Object System for Open-World Instance Segmentation With Object PriorsECCV 2024
- Pointly-Supervised Instance SegmentationCVPR 2022
- Weakly- and Semi-Supervised Panoptic SegmentationECCV 2018
- BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic SegmentationICCV 2015
- ShapeMask: Learning to Segment Novel Objects by Refining Shape PriorsICCV 2019
- SIM: Semantic-Aware Instance Mask Generation for Box-Supervised Instance SegmentationCVPR 2023
- Vision Transformers Are Good Mask Auto-LabelersCVPR 2023
- Semi-Supervised Semantic Image Segmentation With Self-Correcting NetworksCVPR 2020
- Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding BoxesECCV 2022
- Prior to Segment: Foreground Cues for Weakly Annotated Classes in Partially Supervised Instance SegmentationICCV 2021
- Weakly- and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image SegmentationICCV 2015
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic SegmentationCVPR 2024
- Learning Saliency Propagation for Semi-Supervised Instance SegmentationCVPR 2020
Related open problems
Semantic & Instance Segmentation
Overcoming the Mask Proposal Recall Ceiling in Open-Vocabulary Segmentation
Current open-vocabulary and weakly supervised segmentation pipelines decouple class-agnostic mask proposal generation from segment classification, freezing an upstream proposal model like SAM, FreeSOLO, or Mask2Former. Because the downstream classifier operates solely on candidate masks, segmentation accuracy on novel classes and sub-object regions is strictly bounded by the proposal generator's recall, creating a ceiling that downstream improvements cannot fix. Failures to isolate novel or fine-grained instances at the proposal stage permanently propagate through the pipeline without any mechanism for recovery.
Semantic & Instance Segmentation
Benchmarking and Bridging Idealized Preconditions in Promptable Segmentation
Current promptable segmentation models systematically rely on idealized preconditions during evaluation and training, including ground-truth-derived prompts (points or tight boxes extracted directly from GT masks), dense instance annotations, and perfect spatial feature alignment. In real-world interactive deployment, prompts originate from noisy human clicks, coarse bounding boxes, or imperfect auxiliary detectors, leading to uncharacterized performance degradation. Furthermore, models optimized exclusively on point prompts fail to maintain performance across alternative prompt modalities or without high-resolution decoders. Consequently, reported benchmark gains reflect idealized prompting setups rather than actual operational capability.
Semantic & Instance Segmentation
Hierarchical Layout and Vector Segmentation Under Incomplete Multi-Level Supervision
Current instance and semantic segmentation methods for structured documents and vector graphics rely on full, multi-tier hierarchical annotations (simultaneously requiring word, line, and paragraph labels) and fail when vector elements are deeply nested, tangled, or overlapping. Because multi-level hierarchical annotations are significantly more expensive than standard single-level bounding boxes, deploying segmentation pipelines to diverse real-world documents remains cost-prohibitive. Furthermore, the inability to parse overlapping and nested primitives prevents reliable digitization of technical figures, CAD drawings, and complex vector assets.