Open Problems

Learning with Noisy & Partial Labels

Validation-Free Hyperparameter Selection for Robust Learning Pipelines

Effect to explainPartly addressed
Strong candidate · 4/5 runs49 papers report this39% from 2025+

Generated automatically from the limitations stated in 49 papers (CVPR, ICML, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Across these works, the reported gains hinge on hyperparameters whose optima shift per dataset, noise level, class prior, or architecture, and which are chosen by grid search on held-out clean validation data or expensive ablation. This blocks plug-and-play deployment: a practitioner cannot know whether a method's claimed advantage survives at untuned settings, and comparisons between methods are confounded by unequal tuning budgets. Until hyperparameter settings can be selected from the training data itself (or shown to be inert), robust-learning methods remain fragile recipes rather than tools.

Why it matters

Methods whose hyperparameters auto-adapt to noise level, class prior, and dataset become deployable without validation data, and reported gains become interpretable as robust improvements rather than tuning artifacts.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Hyperparameter sensitivity audits at scale. Re-run a battery of published robust-learning methods across their reported hyperparameter grids on standard benchmarks, measuring the performance variance attributable to tuning versus the claimed improvement over baselines. Deliverable: a public sensitivity leaderboard quantifying which methods' gains exceed their tuning sensitivity.

  2. 2

    Self-tuning criteria from training dynamics. Develop and evaluate estimators (e.g., based on label-noise level estimation, per-sample confidence distributions, or loss-curvature signals) that set key hyperparameters (thresholds, loss weights, warm-up lengths) from observable training statistics alone. Measured by: gap between auto-selected and oracle grid-searched performance across datasets and noise regimes.

  3. 3

    Robust-by-construction hyperparameter intervals. For methods with narrow operating ranges (e.g., candidate-set thresholds), reformulate the objective so performance degrades gracefully outside the tuned range, and measure worst-case performance across the reported grid rather than best-case.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Auto-tuning signals may themselves need dataset-specific calibration, recreating the problem one level up; and worst-case-over-grid evaluation may simply show that every method is fragile, yielding a negative result without a constructive path forward.

Sub-problems

  • Benchmarking and Resolving Hyperparameter Sensitivity in Target-Free Domain Adaptation

    Current domain adaptation and generalization algorithms introduce numerous sensitive loss weights, filtering thresholds, and momentum parameters that are routinely tuned using target-domain validation labels. In realistic deployment, target labels are inaccessible, making this selection protocol a violation of the problem setting. Consequently, published performance gains frequently collapse when hyperparameters cannot be oracle-tuned on the unseen target distribution.

  • Validation-Free Hyperparameter Selection for Parameter-Efficient Model Merging

    Current parameter-efficient adaptation and model merging methods introduce structural hyperparameters—such as scaling coefficients ($\alpha, \beta, \lambda$) and component counts ($n, k, K$)—that must be tuned against target validation sets. In zero-shot transfer, out-of-distribution deployment, and cross-task model merging, labeled target validation data is absent by definition, rendering these methods unusable or sensitive to arbitrary defaults. Without validation-free selection criteria, the practical utility of modular model combination remains dependent on the very target-task supervision it seeks to avoid.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 49 papers

Nearest existing work

Related open problems

Learning with Noisy & Partial Labels

Barrier to removePartly addressed

Eliminating the Auxiliary Clean-Label Anchor Required to Bootstrap Training under Imperfect Supervision

Across weak-supervision methods — noisy labels, partial labels, single positives, model merging, node classification — the same structural dependency recurs: the method only works if some external supervised anchor exists, whether a verified clean reference set, a small labeled validation split, class-frequency priors from a labeled pool, or a warm-up model trained on trustworthy data. In the regime where imperfect labeling actually occurs (annotation too expensive or impossible), that anchor is exactly what is missing, so these methods cannot be deployed there at all. The result is a circular dependency: robust learning from weak labels presupposes the labels one is trying to avoid collecting. Today there is no principled way to replace the anchor with signals derivable from the weak data itself.

Strong candidate · 4/5 runs63 papers report this35% from 2025+

Learning with Noisy & Partial Labels

Effect to explainOpen

Hyperparameter-Free Generalization: Benchmarking and Predicting Sensitivity of New Hyperparameters Across Tasks

Nearly every new method in this space ships with two to five new hyperparameters whose optimal values are dataset- or architecture-dependent, and the reported results are only achievable after per-task grid search on held-out validation data. This inflates effective cost, undermines plug-and-play claims, and silently biases the literature toward methods whose authors had tuning budget. Worse, when validation data is scarce (OOD, few-shot, cross-domain), the reported optima may not exist at transfer time at all — yet no one measures how much of a method's claimed gain survives under zero-tuning conditions.

Possible candidate · 1/2 runs23 papers report this27% from 2025+

Learning with Noisy & Partial Labels

Effect to explainPartly addressed

Robust Learning under Incomplete, Evolving, and Annotator-Heterogeneous Label Spaces

Nearly every label-learning method in circulation assumes a closed world: the class set is known a priori, fixed across training and deployment, fully covered by the annotation, and interpreted identically by every annotator. When any of these fail — a class missing from the taxonomy, new classes arriving in a stream, annotators with different accuracies or selection biases, or legacy datasets with only collapsed majority labels — the methods either fail silently or degrade without diagnostic. Today there is no principled way to even measure how much of a model's error comes from label-space misspecification versus model capacity, so practitioners cannot tell whether to fix the taxonomy, re-annotate, or retrain.

Possible candidate · 2/5 runs13 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.