Open Problems

Learning with Noisy & Partial Labels

Robust Learning under Incomplete, Evolving, and Annotator-Heterogeneous Label Spaces

Effect to explainPartly addressed
Possible candidate · 2/5 runs13 papers report this33% from 2025+

Generated automatically from the limitations stated in 13 papers (NeurIPS, EMNLP, ICML), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Nearly every label-learning method in circulation assumes a closed world: the class set is known a priori, fixed across training and deployment, fully covered by the annotation, and interpreted identically by every annotator. When any of these fail — a class missing from the taxonomy, new classes arriving in a stream, annotators with different accuracies or selection biases, or legacy datasets with only collapsed majority labels — the methods either fail silently or degrade without diagnostic. Today there is no principled way to even measure how much of a model's error comes from label-space misspecification versus model capacity, so practitioners cannot tell whether to fix the taxonomy, re-annotate, or retrain.

Why it matters

Learning systems that degrade gracefully instead of silently when the label world shifts, and a diagnostic that tells practitioners whether their annotation protocol or their model is the bottleneck.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Build a misspecification stress-test harness: take standard benchmark datasets, systematically inject the three failure modes (delete classes, stream in novel classes, simulate annotators with varying skill and class-selection bias), and measure the error decomposition of representative noisy/partial-label methods as each assumption is violated. Deliverable within a month: a quantified sensitivity curve per method.

  2. 2

    Develop an estimator that treats the effective label set as latent and partially observed: infer which classes are present but unnamed from the structure of annotator disagreements, and validate by measuring recovery of held-out held-back classes on datasets where the full taxonomy is known.

  3. 3

    Extend the estimator to the online setting with an explicit change-point signal for "a new class has appeared," measuring detection delay and false-alarm rate on streaming benchmarks.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Annotator disagreement patterns may simply not carry enough signal to distinguish a genuinely novel latent class from ordinary annotator confusion, in which case the estimator recovers nothing beyond what a confusion matrix already shows.

Sub-problems

  • Learning Subjective Emotion Distributions Without Per-Annotator Identifiers

    Current perspectivist and annotator-aware emotion models require disaggregated, multi-annotator raw votes tied to specific annotator identifiers to model subjective disagreement. However, the majority of existing large-scale emotion and sentiment benchmarks only release aggregated summary statistics such as mean intensity scores or majority-vote labels. Consequently, modern perspectivist techniques cannot be trained on or applied to legacy benchmark corpora, leaving supervised models to treat genuine human subjectivity as label noise and cap predictive performance.

  • Open-World Partial and Noisy Label Learning with Dynamic Class Discovery

    Current weakly and partially supervised learning methods strictly presuppose a closed world where all semantic classes are known a priori and label distributions remain static. When applied to real-world streaming data where novel categories continuously emerge without pre-assigned identifiers or prior frequency counts, these models force unseen classes into known categories, corrupting existing representations. As a consequence, weakly supervised systems cannot be deployed in open-world, long-running environments without periodic, expensive complete retraining from scratch on globally curated label taxonomies.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 13 papers

Nearest existing work

Related open problems

Learning with Noisy & Partial Labels

Barrier to removePartly addressed

Eliminating the Auxiliary Clean-Label Anchor Required to Bootstrap Training under Imperfect Supervision

Across weak-supervision methods — noisy labels, partial labels, single positives, model merging, node classification — the same structural dependency recurs: the method only works if some external supervised anchor exists, whether a verified clean reference set, a small labeled validation split, class-frequency priors from a labeled pool, or a warm-up model trained on trustworthy data. In the regime where imperfect labeling actually occurs (annotation too expensive or impossible), that anchor is exactly what is missing, so these methods cannot be deployed there at all. The result is a circular dependency: robust learning from weak labels presupposes the labels one is trying to avoid collecting. Today there is no principled way to replace the anchor with signals derivable from the weak data itself.

Strong candidate · 4/5 runs63 papers report this35% from 2025+

Learning with Noisy & Partial Labels

Effect to explainPartly addressed

Validation-Free Hyperparameter Selection for Robust Learning Pipelines

Across these works, the reported gains hinge on hyperparameters whose optima shift per dataset, noise level, class prior, or architecture, and which are chosen by grid search on held-out clean validation data or expensive ablation. This blocks plug-and-play deployment: a practitioner cannot know whether a method's claimed advantage survives at untuned settings, and comparisons between methods are confounded by unequal tuning budgets. Until hyperparameter settings can be selected from the training data itself (or shown to be inert), robust-learning methods remain fragile recipes rather than tools.

Strong candidate · 4/5 runs49 papers report this39% from 2025+

Learning with Noisy & Partial Labels

Effect to explainOpen

Hyperparameter-Free Generalization: Benchmarking and Predicting Sensitivity of New Hyperparameters Across Tasks

Nearly every new method in this space ships with two to five new hyperparameters whose optimal values are dataset- or architecture-dependent, and the reported results are only achievable after per-task grid search on held-out validation data. This inflates effective cost, undermines plug-and-play claims, and silently biases the literature toward methods whose authors had tuning budget. Worse, when validation data is scarce (OOD, few-shot, cross-domain), the reported optima may not exist at transfer time at all — yet no one measures how much of a method's claimed gain survives under zero-tuning conditions.

Possible candidate · 1/2 runs23 papers report this27% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.