Open Problems

Continual Learning & Catastrophic Forgetting

Characterizing and Reducing Pre-Trained ViT Dependence in Transfer-Based Methods

Scope to testOpen
Possible candidate · 3/5 runs20 papers report this78% from 2025+

Generated automatically from the limitations stated in 20 papers (CVPR, NeurIPS, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

A wide range of methods built on frozen ImageNet-pretrained ViT features—whether for adapting to new tasks, constraining representations, or detecting anomalies—inherit an unexamined dependency: their guarantees hold only when the backbone's feature space is already good. Today nobody knows whether these methods degrade gracefully with weaker pretraining, generalize to domains far from ImageNet, or transfer to non-transformer architectures, because the dependence is never ablated. The consequence is that reported gains may be properties of the backbone, not of the methods themselves, and the methods are silently inapplicable wherever a strong ImageNet ViT does not exist.

Why it matters

A principled account of when feature-frozen methods can be deployed outside the ImageNet-ViT regime, and design guidance for methods that degrade gracefully rather than collapse when pretraining is weak or absent.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Backbone-degradation ablation suite: take 3–5 representative feature-frozen methods (one adapter-based continual learning method, one geometric/frozen-feature method, one anomaly detection method) and re-evaluate them across a controlled pretraining gradient—ImageNet-21k, ImageNet-1k, small-scale supervised, self-supervised MAE, random init—holding the method fixed and measuring accuracy, forgetting, and AUROC as a function of backbone quality. Deliverable: a quantitative "degradation curve" per method showing where each breaks.

  2. 2

    Cross-architecture transfer test: rerun the same methods on ConvNeXt, Swin, and ResNet backbones with matched pretraining data, measuring whether failures are transformer-specific or feature-quality-specific. Measured: rank correlation between backbone linear-probe accuracy and downstream method performance.

  3. 3

    Domain-distance study: evaluate on domains far from ImageNet (medical, remote sensing, audio spectrograms) with domain-matched pretraining, to separate "the method needs good features" from "the method needs ImageNet features specifically."

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

The finding could be entirely unsurprising—everyone may already know performance tracks backbone quality, and the degradation curves may show smooth, predictable declines with no interesting structure or collapse points worth reporting.

Sub-problems

  • Continual Learning Under Backbone and Pre-Training Domain Shift

    Current top-performing continual learning methods, especially prompt- and adapter-based approaches, rely almost entirely on frozen Vision Transformer backbones pre-trained on ImageNet. In real-world deployments where the target stream lies far outside natural image distributions (e.g., medical imaging, remote sensing) or where compute constraints necessitate lightweight non-ViT architectures, these methods lack pre-separated feature spaces and their performance guarantees evaporate. Consequently, the field cannot distinguish whether contemporary continual learning algorithms prevent catastrophic forgetting through robust plastic-stable dynamics or merely exploit the static separability of upstream pre-trained representations.

  • Anomaly Detection and Representation Adaptation Without Pre-Trained Foundation Backbones

    Modern visual anomaly and out-of-distribution detection methods rely almost entirely on frozen features extracted from foundation models pre-trained on ImageNet. When applied to domains whose visual primitives are absent from standard pre-training datasets—such as specialized industrial inspection, semiconductor wafer maps, or specialized biomedical imaging—these methods degrade significantly or fail entirely. Because standard methods do not fine-tune backbones on normal-only target data due to feature collapse, practitioners in non-natural image domains are blocked from using state-of-the-art anomaly detection pipelines.

Evidence

Each paper's own statement of the limitation, verbatim.

Show all 20 papers

Nearest existing work

Related open problems

Continual Learning & Catastrophic Forgetting

Scope to testPartly addressed

A Standardized Transfer-Validation Benchmark for Methods Validated Only on Small-Scale Image Classification

Nearly every training-time method in the modern literature—continual learning, distillation, unlearning, robustness, federated aggregation—is validated exclusively on CIFAR/ImageNet-subset image classification with ResNet-18/50-class backbones, and each paper explicitly defers scale, architecture, and modality transfer to future work. Because every paper defers this simultaneously, the field has no evidence base at all for whether any of these methods survive the shift to transformers, higher resolutions, dense tasks, or non-vision data; practitioners choosing between methods are choosing based on measurements made in a regime none of them deploy in. The barrier is not one method's limitation but a collective, unpriced evaluation debt: nobody can tell which small-scale conclusions are regime-robust and which are artifacts of ResNet-18 on 32×32 images.

Possible candidate · 3/5 runs152 papers report this44% from 2025+

Continual Learning & Catastrophic Forgetting

UnclassifiedOpen

Sub-Quadratic Covariance Sketching for Exemplar-Free Continual Learning

Exemplar-free continual learning methods increasingly rely on storing per-class feature statistics (prototypes and full covariance matrices) to generate pseudo-replay features or parameterize Mahalanobis classifiers without retaining raw inputs. However, storing a dense $D \times D$ covariance matrix for each of $C$ classes incurs $\mathcal{O}(C \cdot D^2)$ storage, which quickly exceeds device memory when scaling to large label spaces ($C \ge 1000$) or modern high-dimensional embeddings ($D \ge 1024$). Practitioners currently resort to naive diagonal approximations that discard inter-feature correlations and cause severe classification degradation under continual distribution shifts. As a result, covariance-based exemplar-free learning is structurally blocked from scaling to realistic large-vocabulary benchmarks on memory-constrained hardware.

Possible candidate · 3/5 runs6 papers report this50% from 2025+

Continual Learning & Catastrophic Forgetting

Barrier to removePartly addressed

Task-Agnostic Inference for Modular and Masked Continual Learning Architectures

Modular continual learning methods prevent catastrophic forgetting by allocating task-specific masks, sub-networks, or adaptive parameters during training. However, these methods structurally depend on having ground-truth task identifiers provided at test time to select the corresponding parameters. In practical deployments, incoming inputs arrive without task metadata, rendering task-dependent parameter masking unusable in class-incremental or task-agnostic settings. Without a reliable mechanism to resolve task identity or route inputs dynamically at test time, parameter-isolation architectures remain restricted to artificial task-incremental benchmarks.

Possible candidate · 2/5 runs3 papers report this0% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.