Open Problems

Learning with Noisy & Partial Labels

Hyperparameter-Free Generalization: Benchmarking and Predicting Sensitivity of New Hyperparameters Across Tasks

Effect to explainOpen
Possible candidate · 1/2 runs23 papers report this27% from 2025+

Generated automatically from the limitations stated in 23 papers (CVPR, ACL, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Nearly every new method in this space ships with two to five new hyperparameters whose optimal values are dataset- or architecture-dependent, and the reported results are only achievable after per-task grid search on held-out validation data. This inflates effective cost, undermines plug-and-play claims, and silently biases the literature toward methods whose authors had tuning budget. Worse, when validation data is scarce (OOD, few-shot, cross-domain), the reported optima may not exist at transfer time at all — yet no one measures how much of a method's claimed gain survives under zero-tuning conditions.

Why it matters

A tuning-free evaluation standard that reveals which methods' gains are real versus artifacts of per-dataset search, and enables principled deployment in validation-scarce settings (OOD, few-shot, cross-domain transfer).

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Build a "tuning sensitivity audit" protocol: for 15–20 methods with known new hyperparameters, re-evaluate each across ≥10 datasets under (a) default/fixed values, (b) per-dataset tuning, and (c) transfer of tuned values across datasets and architectures. Measure the gain attributable to tuning vs. the method itself, producing a public leaderboard of hyperparameter sensitivity. Startable immediately with published code.

  2. 2

    Develop sensitivity predictors: fit simple meta-models over dataset statistics (size, noise level, label quality, domain shift) to predict near-optimal hyperparameter values without validation data; evaluate on the transfer setting from (1) by measuring regret relative to tuned optima.

  3. 3

    Formalize "default robustness" as an evaluation criterion — an interval around each hyperparameter within which performance degrades <1% — and test whether existing methods can be re-tuned once to a robust default that works everywhere, reporting where no such default exists.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The sensitivity audit may simply confirm what everyone already anecdotally knows ("tuning matters") without yielding transferable predictors, and meta-models may fail because optimal hyperparameters depend on interactions too specific to dataset statistics to generalize.

Sub-problems

  • Validation-Free Hyperparameter Estimation for Model Merging and Modular Adaptation

    Current parameter-efficient adaptation and model merging methods rely on tuning structural hyperparameters—such as scaling coefficients ($\lambda, \alpha$), rank thresholds ($k$), and expert counts ($K$)—using held-out labeled validation data. In zero-shot deployment, out-of-distribution transfer, and decentralized adaptation, such labeled validation splits do not exist. Practitioners are consequently forced to rely on arbitrary default hyperparameters, which frequently leads to catastrophic interference, sub-optimal skill combination, or severe overfitting.

  • Self-Regulating Auxiliary Hyperparameters for Sample-Constrained Bayesian Optimization

    Bayesian optimization is designed for sample-efficient optimization of expensive black-box functions where evaluations are strictly budget-constrained. When advanced BO methods introduce internal auxiliary hyperparameters (such as decay rates, coupling weights, and softmax temperatures) that require manual tuning or grid search, the meta-tuning cost directly defeats the premise of sample-efficient optimization. In single-run or physical experimental campaigns, practitioners cannot afford pilot trials to calibrate the optimizer itself. Consequently, advanced BO formulations remain fragile or unusable outside the specific benchmark suites on which their internal parameters were tuned.

  • Target-Free Hyperparameter Selection for Few-Shot and Zero-Shot Transfer

    Few-shot and zero-shot algorithms are highly sensitive to hyperparameters like activation thresholds, temperature scalings, and projection dimensions, but standard workflows tune these against labeled target-domain validation sets. In genuine zero-shot and extreme few-shot deployments, target validation labels are unavailable by definition, forcing practitioners to use arbitrary literature defaults that severely degrade downstream performance. This creates a structural evaluation gap where reported benchmarks implicitly rely on label leakage to achieve reported performance.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 23 papers

Nearest existing work

Related open problems

Learning with Noisy & Partial Labels

Barrier to removePartly addressed

Eliminating the Auxiliary Clean-Label Anchor Required to Bootstrap Training under Imperfect Supervision

Across weak-supervision methods — noisy labels, partial labels, single positives, model merging, node classification — the same structural dependency recurs: the method only works if some external supervised anchor exists, whether a verified clean reference set, a small labeled validation split, class-frequency priors from a labeled pool, or a warm-up model trained on trustworthy data. In the regime where imperfect labeling actually occurs (annotation too expensive or impossible), that anchor is exactly what is missing, so these methods cannot be deployed there at all. The result is a circular dependency: robust learning from weak labels presupposes the labels one is trying to avoid collecting. Today there is no principled way to replace the anchor with signals derivable from the weak data itself.

Strong candidate · 4/5 runs63 papers report this35% from 2025+

Learning with Noisy & Partial Labels

Effect to explainPartly addressed

Validation-Free Hyperparameter Selection for Robust Learning Pipelines

Across these works, the reported gains hinge on hyperparameters whose optima shift per dataset, noise level, class prior, or architecture, and which are chosen by grid search on held-out clean validation data or expensive ablation. This blocks plug-and-play deployment: a practitioner cannot know whether a method's claimed advantage survives at untuned settings, and comparisons between methods are confounded by unequal tuning budgets. Until hyperparameter settings can be selected from the training data itself (or shown to be inert), robust-learning methods remain fragile recipes rather than tools.

Strong candidate · 4/5 runs49 papers report this39% from 2025+

Learning with Noisy & Partial Labels

Effect to explainPartly addressed

Robust Learning under Incomplete, Evolving, and Annotator-Heterogeneous Label Spaces

Nearly every label-learning method in circulation assumes a closed world: the class set is known a priori, fixed across training and deployment, fully covered by the annotation, and interpreted identically by every annotator. When any of these fail — a class missing from the taxonomy, new classes arriving in a stream, annotators with different accuracies or selection biases, or legacy datasets with only collapsed majority labels — the methods either fail silently or degrade without diagnostic. Today there is no principled way to even measure how much of a model's error comes from label-space misspecification versus model capacity, so practitioners cannot tell whether to fix the taxonomy, re-annotate, or retrain.

Possible candidate · 2/5 runs13 papers report this33% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.