Open Problems

Continual Learning & Catastrophic Forgetting

A Standardized Transfer-Validation Benchmark for Methods Validated Only on Small-Scale Image Classification

Scope to testPartly addressed
Possible candidate · 3/5 runs152 papers report this44% from 2025+

Generated automatically from the limitations stated in 152 papers (CVPR, ICML, NeurIPS), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Nearly every training-time method in the modern literature—continual learning, distillation, unlearning, robustness, federated aggregation—is validated exclusively on CIFAR/ImageNet-subset image classification with ResNet-18/50-class backbones, and each paper explicitly defers scale, architecture, and modality transfer to future work. Because every paper defers this simultaneously, the field has no evidence base at all for whether any of these methods survive the shift to transformers, higher resolutions, dense tasks, or non-vision data; practitioners choosing between methods are choosing based on measurements made in a regime none of them deploy in. The barrier is not one method's limitation but a collective, unpriced evaluation debt: nobody can tell which small-scale conclusions are regime-robust and which are artifacts of ResNet-18 on 32×32 images.

Why it matters

For the first time, claims like "works at scale" or "architecture-agnostic" become testable and comparable across subfields, and method selection stops resting on 32×32 ResNet evidence alone.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Build a fixed "transfer grid" — a small, frozen set of evaluation points spanning (backbone: ResNet-18, ViT-Ti/B; scale: CIFAR-100, ImageNet-1k; task: classification, one dense task; modality: vision, text) — and port 5–8 representative open-source methods from different training-time families onto it, measuring exactly which ones degrade, fail to train, or require re-tuning at each grid point. Deliverable: a public leaderboard with per-cell reproduction instructions and compute budgets.

  2. 2

    Run a systematic "regime sensitivity" study: for each method, quantify the gap between its reported small-scale headline result and its result under backbone swap and scale-up, holding hyperparameters fixed vs. re-tuned, to separate intrinsic fragility from mere tuning burden. Measured as a transfer-degradation score per method.

  3. 3

    Distill the grid results into practitioner guidance: which classes of method (e.g., those depending on feature statistics vs. loss geometry) transfer predictably, tested by predicting held-out grid cells from observed degradation patterns.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The grid's own coverage is finite, so it risks becoming one more small benchmark that methods overfit to, and compute costs may force most cells to use reduced resolutions — recreating the very regime the project exists to escape.

Sub-problems

  • Cross-Architecture and Cross-Task Generalisation of Neural Pruning and NAS Methods

    A vast body of neural architecture search (NAS) and structural pruning algorithms is evaluated exclusively on standard convolutional backbones (e.g., ResNet, MobileNet) on CIFAR and ImageNet classification. Because architectural primitives (e.g., self-attention vs. spatial convolutions) and loss landscapes differ substantially across modalities and tasks, it is currently unknown whether these methods generalize, degrade, or completely fail when applied to Transformers, object detection backbones, or language tasks. Without systematic cross-paradigm evaluation, researchers and practitioners cannot determine whether published NAS/pruning performance gains reflect general compression principles or narrow overfitting to standard 2D-CNN benchmarks.

  • Benchmarking Dataset Distillation Across Image Resolutions, Dataset Scales, and Model Families

    Dataset distillation methods are almost exclusively developed and evaluated on low-resolution toy benchmarks (CIFAR, Tiny-ImageNet at 32x32) using shallow 3-layer ConvNets. Consequently, it is unknown whether the relative performance of different distillation paradigms (e.g., gradient matching, trajectory matching, distribution matching) holds at high resolutions, on full ImageNet-1K scale, or when transferring to modern vision architectures like Vision Transformers. Practitioners cannot rely on synthetic datasets for real-world downstream training or architecture search because performance rankings established on toy regimes may be artifacts of shallow convolutional surrogates.

  • Cross-Architecture and Scale Robustness of Continual Learning Algorithms

    The continual learning literature currently evaluates algorithms almost exclusively on small convolutional backbones (primarily ResNet-18) and downsampled vision benchmarks (CIFAR-100, Tiny-ImageNet). It is unknown whether canonical algorithmic rankings—such as the relative efficacy of experience replay, weight regularization, and parameter isolation—persist when transitioning to modern transformer backbones, full-scale image datasets, or non-vision modalities. Without systematic cross-architecture evaluation, continual learning research risks optimizing techniques for artifacts specific to low-capacity CNNs on small-scale vision splits rather than addressing catastrophic forgetting in modern foundational architectures.

  • A Systematic Generalization Benchmark for Dataset Distillation Across Dense Prediction and Non-Visual Modalities

    Dataset distillation algorithms compress large training sets into compact synthetic samples, but current literature assesses them almost exclusively on image classification at low resolutions (≤128×128). Dense visual tasks such as object detection and semantic segmentation require fine-grained spatial and contextual preservation, while text and graph modalities require maintaining discrete structural semantics. Without empirical evaluation across these settings, practitioners cannot know whether current distillation objectives generalize to structured tasks or whether synthetic samples capture only global class-discriminative shortcuts.

  • Cross-Modality and Large-Scale Empirical Robustness of Federated Learning Algorithms

    The vast majority of federated learning (FL) algorithms are evaluated exclusively on small vision datasets (such as CIFAR-10/100 and MNIST) using small CNN backbones. As a result, it is unknown whether established FL aggregation, regularization, and personalization techniques remain effective when applied to other modalities (natural language, tabular, audio) or modern large architectures (transformers, pre-trained backbones). Practitioners and researchers cannot reliably predict which algorithmic components fail or generalize beyond small-scale computer vision benchmarks.

  • Benchmarking Noisy and Partial Label Learning Across Model Architectures, Scale, and Modalities

    The vast majority of algorithms for learning with noisy and partial labels are validated exclusively on low-resolution image benchmarks (CIFAR-10/100, TinyImageNet) using small convolutional backbones like ResNet-18. Consequently, practitioners have no reliable evidence on whether these loss corrections, sample-selection heuristics, or label-disambiguation schemes retain their relative performance on modern Transformer architectures, larger dataset scales, or non-vision modalities (NLP and tabular). Without cross-setting evaluation, methods tailored to small ResNets risk being brittle artifacts of low-capacity, small-resolution image regimes.

  • Systematic Evaluation of Adversarial Robustness and Transferability Across Architecture Families and Dataset Scales

    Adversarial attack and defense mechanisms are predominantly validated on legacy convolutional neural networks (CNNs) and small 1,000-image ImageNet subsets. Because these methods are rarely evaluated on modern non-convolutional backbones such as Vision Transformers or across full-scale datasets and varying perturbation norms, their true cross-model efficacy remains unknown. Consequently, practitioners deploying defenses on modern vision architectures have no empirical guarantees on whether reported robustness gains transfer or cause unacceptable natural accuracy degradation.

  • Cross-Task and Cross-Modal Robustness Evaluation of Self-Supervised Representations

    Core empirical findings and design choices in self-supervised and contrastive learning are almost exclusively established on ImageNet classification. It is unknown whether the standard inductive biases, augmentations, and objective functions that optimize classification transfer effectively to dense visual tasks (e.g., detection, segmentation) or non-vision modalities (e.g., audio, text). Without systematic cross-task and cross-modal evaluation, the community risks standardizing architectures and pre-training recipes that overfit to categorical image benchmarking while failing in broader operational settings.

  • Cross-Domain and Scale Generalization of Continual Learning Methods

    Current continual learning techniques—such as rehearsal buffers, regularizers, and architecture-expansion methods—are almost exclusively evaluated on small, low-resolution image classification datasets (e.g., CIFAR-10/100, TinyImageNet) using small CNN backbones. Practitioners cannot determine whether these methods preserve performance when applied to dense vision tasks (detection, segmentation), language tasks, modern transformer architectures, or long-horizon task streams. As a result, deployment in real-world continuous adaptation pipelines remains blocked because algorithmic performance rankings derived from toy classification fail to transfer predictably to realistic settings.

  • Empirical Generalization of Adversarial and Backdoor Attacks Across Modern Vision Architectures and Scaled Benchmarks

    The vast majority of published claims regarding backdoor attacks, defensive unlearning, and adversarial robustness rely exclusively on low-resolution datasets (e.g., CIFAR-10/100, GTSRB) evaluated on small CNN backbones (predominantly ResNet-18). It is currently unknown whether these attack success rates, perturbation budgets, and defensive mechanisms persist when applied to modern architectural paradigms (e.g., Vision Transformers, ConvNeXt) or larger-scale, higher-resolution tasks. As a result, security assurances and vulnerability assessments in real-world computer vision deployments remain ungrounded in empirical evidence.

  • Cross-Scale and Cross-Modal Robustness Benchmarking for Machine Unlearning

    Current machine unlearning algorithms are evaluated almost exclusively on small vision datasets (e.g., CIFAR-10/100, TinyImageNet) using compact models (e.g., ResNet-18) at low removal ratios. Because unlearning dynamics depend heavily on parameter overcapacity and representation geometry, empirical efficacy on toy classification cannot predict behavior on realistic scales or other modalities. Practitioners cannot deploy unlearning algorithms to fulfill legal or privacy compliance on full-scale architectures because it remains unknown which methods maintain accuracy and forget guarantees outside of small-scale vision.

  • Cross-Paradigm and Scale Generalisation Benchmarks for Dataset Distillation

    Dataset distillation methods are almost exclusively evaluated on small-scale datasets (CIFAR-10/100, Tiny-ImageNet) and homogeneous small ConvNet probe architectures. It is entirely unknown whether distilled synthetic data retains utility when transferred to Vision Transformers, modern hybrid backbones, or higher-resolution regimes with large capacity mismatches. Without systematic cross-architecture evaluation, distillation methods risk overfitting to the inductive biases of shallow ConvNets and failing in realistic downstream training pipelines.

  • Cross-Task Evaluation and Adaptation of Machine Unlearning Beyond Image Classification

    Current machine unlearning algorithms are designed and validated almost exclusively on image classification benchmarks. It remains empirically unknown whether standard unlearning mechanisms—such as influence function approximations, gradient ascent, or Fisher-guided parameter scrubbing—generalise to structured vision tasks like object detection and semantic segmentation. Without systematic evaluation, practitioners cannot determine whether existing unlearning methods corrupt spatial representations, fail to erase localised object concepts, or cause catastrophic utility loss on dense prediction pipelines.

  • Cross-Architecture and Cross-Task Robustness Evaluation of Federated Learning Methods

    Current federated learning methods for robustness and client heterogeneity are evaluated almost exclusively on standard convolutional backbones (primarily ResNet-18/50) across image classification datasets. As a consequence, it is unknown whether the mechanisms and empirical gains reported in the literature depend on CNN-specific properties (such as batch normalization statistics or localized feature representations) or if they hold across modern architectures like Vision Transformers and dense prediction tasks like segmentation and detection. Deploying existing FL techniques into production systems using non-CNN backbones or non-classification tasks carries unquantified risks of performance degradation and instability.

  • Systematic Cross-Architecture and High-Resolution Benchmark for Dataset Distillation

    Dataset distillation methods are almost exclusively evaluated on tiny, low-resolution datasets (such as CIFAR-10 and MNIST) using identical 3-layer convolutional architectures for both synthesis and evaluation. Because existing studies do not evaluate performance when scaling to high-resolution images (such as ImageNet-scale) or transferring synthetic sets to modern deep backbones (such as ResNets and Vision Transformers), it is unknown whether current distillation objectives learn generalizable visual features or exploit inductive biases specific to shallow ConvNets. Without systematic empirical evaluation across diverse resolutions and model families, practitioners cannot determine whether synthetic datasets are viable for downstream training on standard deep learning pipelines.

  • Empirical Scaling and Generalisation Limits of Dataset Distillation Beyond Small Vision Benchmarks

    Current dataset distillation algorithms are evaluated almost exclusively on low-resolution image classification datasets (CIFAR-10/100, Tiny ImageNet) using small ConvNets or shallow ResNets. Consequently, it is unknown whether synthesized datasets transfer effectively to large-scale vision backbones, modern generative models, or language models, despite broad motivational claims. Without cross-scale and cross-architecture evaluation, practitioners cannot determine whether observed distillation gains are algorithmic breakthroughs or artifacts of low-resolution, low-capacity regimes.

  • Empirical Robustness and Scalability Benchmark for Self-Supervised Learning Methods

    Self-supervised learning algorithms are predominantly developed and evaluated on small backbones (ResNet-18) and downscaled subsets (ImageNet-100, STL-10, CIFAR), with gains often depending on setting-specific hyperparameter tuning. Because these methods have not been evaluated across larger backbones (such as Vision Transformers) and full-scale benchmarks (ImageNet-1k) under a unified protocol, it is unknown whether reported algorithmic advantages transfer or are artifacts of small-scale proxy setups. Without systematic multi-scale evaluation, researchers cannot determine which algorithmic mechanisms reliably generalize when scaled.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 152 papers

Nearest existing work

Related open problems

Continual Learning & Catastrophic Forgetting

Scope to testOpen

Characterizing and Reducing Pre-Trained ViT Dependence in Transfer-Based Methods

A wide range of methods built on frozen ImageNet-pretrained ViT features—whether for adapting to new tasks, constraining representations, or detecting anomalies—inherit an unexamined dependency: their guarantees hold only when the backbone's feature space is already good. Today nobody knows whether these methods degrade gracefully with weaker pretraining, generalize to domains far from ImageNet, or transfer to non-transformer architectures, because the dependence is never ablated. The consequence is that reported gains may be properties of the backbone, not of the methods themselves, and the methods are silently inapplicable wherever a strong ImageNet ViT does not exist.

Possible candidate · 3/5 runs20 papers report this78% from 2025+

Continual Learning & Catastrophic Forgetting

UnclassifiedOpen

Sub-Quadratic Covariance Sketching for Exemplar-Free Continual Learning

Exemplar-free continual learning methods increasingly rely on storing per-class feature statistics (prototypes and full covariance matrices) to generate pseudo-replay features or parameterize Mahalanobis classifiers without retaining raw inputs. However, storing a dense $D \times D$ covariance matrix for each of $C$ classes incurs $\mathcal{O}(C \cdot D^2)$ storage, which quickly exceeds device memory when scaling to large label spaces ($C \ge 1000$) or modern high-dimensional embeddings ($D \ge 1024$). Practitioners currently resort to naive diagonal approximations that discard inter-feature correlations and cause severe classification degradation under continual distribution shifts. As a result, covariance-based exemplar-free learning is structurally blocked from scaling to realistic large-vocabulary benchmarks on memory-constrained hardware.

Possible candidate · 3/5 runs6 papers report this50% from 2025+

Continual Learning & Catastrophic Forgetting

Barrier to removePartly addressed

Task-Agnostic Inference for Modular and Masked Continual Learning Architectures

Modular continual learning methods prevent catastrophic forgetting by allocating task-specific masks, sub-networks, or adaptive parameters during training. However, these methods structurally depend on having ground-truth task identifiers provided at test time to select the corresponding parameters. In practical deployments, incoming inputs arrive without task metadata, rendering task-dependent parameter masking unusable in class-incremental or task-agnostic settings. Without a reliable mechanism to resolve task identity or route inputs dynamically at test time, parameter-isolation architectures remain restricted to artificial task-incremental benchmarks.

Possible candidate · 2/5 runs3 papers report this0% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.