Open Problems

Self-Supervised & Contrastive Representation Learning

A Scaling-Law Characterization of Regularization and Transfer Benefits That Vanish at Scale

Effect to explainOpen
Strong candidate · 4/5 runs11 papers report this33% from 2025+

Generated automatically from the limitations stated in 11 papers (ICCV, ICML, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Many published training-time interventions show solid gains on small models, small batches, and small datasets, yet shrink toward zero as any of these scale up. Because each paper reports only its own endpoints, the community cannot tell whether a given benefit decays with scale or was an artifact of the small regime — leading to wasted adoption and redundant re-testing. A controlled characterization of this decay, with predictors of when a benefit will survive, is what is currently missing.

Why it matters

Researchers could predict from small-scale experiments whether a technique will matter at production scale, making small-scale ablations informative again and preventing both false adoption and false dismissal.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Re-run a small panel of representative interventions (a contrastive regularizer, a distillation loss, a feature-level augmentation) across a factorial grid of model size × batch size × dataset size, and fit decay curves for each gain. Measured: absolute gain as a function of each scale axis, with fitted exponents.

  2. 2

    Test whether the gain scales with a measurable "headroom" quantity — e.g., distance of the baseline from an information-theoretic or optimization bound — and validate whether headroom predicts which regimes retain gains. Measured: correlation between predicted and observed gains on held-out scale points.

  3. 3

    Derive a simple decision rule ("skip this method above N parameters / batch / dataset size") and evaluate its error rate across new methods not in the original panel.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The decay curves may not be consistent across methods — different interventions could fail at scale for unrelated reasons, leaving no shared exponent or predictor to fit.

Sub-problems

  • Diagnosing and Overcoming the Diminishing Returns of Knowledge Distillation at Scale

    Knowledge distillation consistently demonstrates large accuracy gains on small-scale datasets and few-shot regimes, but multiple independent studies show these benefits diminish to near-zero on full-scale benchmarks like ImageNet-1k. As a result, practitioners cannot effectively leverage high-capacity foundation teachers to improve deployment-scale models on standard large datasets, making the compute invested in large teachers unrecoverable at inference time. Without identifying why distillation benefits collapse at scale—whether due to capacity saturation under full supervision, representation mismatch, or suboptimal loss objectives—distillation remains structurally limited to low-data settings.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 11 papers

Nearest existing work

Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.