Self-Supervised & Contrastive Representation Learning
A Scaling-Law Characterization of Regularization and Transfer Benefits That Vanish at Scale
Generated automatically from the limitations stated in 11 papers (ICCV, ICML, ECCV), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Many published training-time interventions show solid gains on small models, small batches, and small datasets, yet shrink toward zero as any of these scale up. Because each paper reports only its own endpoints, the community cannot tell whether a given benefit decays with scale or was an artifact of the small regime — leading to wasted adoption and redundant re-testing. A controlled characterization of this decay, with predictors of when a benefit will survive, is what is currently missing.
Why it matters
Researchers could predict from small-scale experiments whether a technique will matter at production scale, making small-scale ablations informative again and preventing both false adoption and false dismissal.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Re-run a small panel of representative interventions (a contrastive regularizer, a distillation loss, a feature-level augmentation) across a factorial grid of model size × batch size × dataset size, and fit decay curves for each gain. Measured: absolute gain as a function of each scale axis, with fitted exponents.
- 2
Test whether the gain scales with a measurable "headroom" quantity — e.g., distance of the baseline from an information-theoretic or optimization bound — and validate whether headroom predicts which regimes retain gains. Measured: correlation between predicted and observed gains on held-out scale points.
- 3
Derive a simple decision rule ("skip this method above N parameters / batch / dataset size") and evaluate its error rate across new methods not in the original panel.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
The decay curves may not be consistent across methods — different interventions could fail at scale for unrelated reasons, leaving no shared exponent or predictor to fit.
Sub-problems
Diagnosing and Overcoming the Diminishing Returns of Knowledge Distillation at Scale
Knowledge distillation consistently demonstrates large accuracy gains on small-scale datasets and few-shot regimes, but multiple independent studies show these benefits diminish to near-zero on full-scale benchmarks like ImageNet-1k. As a result, practitioners cannot effectively leverage high-capacity foundation teachers to improve deployment-scale models on standard large datasets, making the compute invested in large teachers unrecoverable at inference time. Without identifying why distillation benefits collapse at scale—whether due to capacity saturation under full supervision, representation mismatch, or suboptimal loss objectives—distillation remains structurally limited to low-data settings.
Evidence
Each paper's own statement of the limitation, verbatim.
- SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher FeaturesAISTATS 2026
Gains in absolute terms are small on some settings (e.g., +0.40 pp on ImageNet-1K 1-bit), so benefits may be marginal for large-scale training
- SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient OptimizationNeurIPS 2025
Gains on large-scale ImageNet are small (+0.34 to +1.88), with the largest gains concentrated on small/fine-grained datasets, so benefit at scale may be limited
- On the Similarities of Embeddings in Contrastive LearningICML 2025
Gains are concentrated in small-batch settings; with large batches the effect diminishes, and ImageNet results are relegated to the appendix.
- Cross-Architecture Distillation Made Simple with Redundancy SuppressionICCV 2025
Gains are less prominent on larger-scale datasets (ImageNet-1k) than CIFAR-100, partly because 2-D features are unused
- ConMix: Contrastive Mixup at Representation Level for Long-tailed Deep ClusteringICLR 2025
On Tiny ImageNet (200 classes, ratio 10), improvements are marginal (16.5 ACC vs. 16.1 BYOL; NMI 34.9 vs. 35.2 SDCLR), suggesting effectiveness degrades with many classes
- Good Teachers Explain: Explanation-Enhanced Knowledge DistillationECCV 2024
Gains shrink substantially at full data scale (ImageNet full data: +0.0 accuracy for KD+e2GD vs. +5.1 at 50 shots), so benefit is mainly in low-data regimes
- Deep Companion Learning: Enhancing Generalization Through Historical ConsistencyECCV 2024
Gains shrink on large-scale ImageNet-1k (0.4–1.1 points) relative to CIFAR-100 (up to 4.1 points), suggesting the regularizer's benefit diminishes with dataset size
- AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift FactorCVPR 2024
Gains shrink as models grow (from +5.2 on ResNet-14 to +1.7 for AdaShift-B on ResNet-101), so the benefit is smaller for large models.
Show all 11 papers
- Semantic Information in Contrastive LearningICCV 2023
Gains over ImageNet counterparts are small in absolute terms (mostly +0.1 to +0.8 mIoU/AP), and some degrade (Swin-S −0.18 on COCO seg)
- Model-Aware Contrastive Learning: Towards Escaping the DilemmasICML 2023
Gains shrink as batch/negative size grows (+2.4 at batch 256 vs +0.8 at 2048 on ImageNet-1K), so benefit is largest only in small-batch regimes
- Cumulative Spatial Knowledge Distillation for Vision TransformersICCV 2023
Absolute gains on larger models are small (CSKD-B +0.4% over DeiT-B on ImageNet-1k)
Nearest existing work
- How to set AdamW’s weight decay as you scale model and dataset sizeICML 2025
- Why Do We Need Weight Decay in Modern Deep Learning?NeurIPS 2024
- Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum LearningACL 2025
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural NetworksNeurIPS 2025
- CompRess: Self-Supervised Learning by Compressing RepresentationsNeurIPS 2020
- Small-scale proxies for large-scale Transformer training instabilitiesICLR 2024
- Beyond neural scaling laws: beating power law scaling via data pruningNeurIPS 2022
- Data Efficient Neural Scaling Law via Model ReusingICML 2023
- Exploring Structural Degradation in Dense Representations for Self-supervised LearningNeurIPS 2025
- AUC-CL: A Batchsize-Robust Framework for Self-Supervised Contrastive Representation LearningICLR 2024
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural NetworksICML 2025
- A Dynamical Model of Neural Scaling LawsICML 2024
- How to Scale Your EMANeurIPS 2023
- Catastrophic Forgetting Meets Negative Transfer: Batch Spectral Shrinkage for Safe Transfer LearningNeurIPS 2019
- Towards Precise Scaling Laws for Video Diffusion TransformersCVPR 2025