Open Problems

Continual Learning & Catastrophic Forgetting

Task-Agnostic Inference for Modular and Masked Continual Learning Architectures

Barrier to removePartly addressed
Possible candidate · 2/5 runs3 papers report this0% from 2025+

Generated automatically from the limitations stated in 3 papers (CVPR, ICML, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.

The problem

Modular continual learning methods prevent catastrophic forgetting by allocating task-specific masks, sub-networks, or adaptive parameters during training. However, these methods structurally depend on having ground-truth task identifiers provided at test time to select the corresponding parameters. In practical deployments, incoming inputs arrive without task metadata, rendering task-dependent parameter masking unusable in class-incremental or task-agnostic settings. Without a reliable mechanism to resolve task identity or route inputs dynamically at test time, parameter-isolation architectures remain restricted to artificial task-incremental benchmarks.

Why it matters

Parameter-isolation and masked continual learning techniques become deployable in class-incremental and unlabelled streaming environments where test inputs lack task identifiers.

Ways to approach it

Prior-work checks are free with an account. Results someone already ran are shown to everyone.

  1. 1

    Benchmark out-of-distribution task routing: implement task-specific density estimators (e.g., Mahalanobis distance or energy-based scores on intermediate representations) alongside frozen task modules, measuring routing accuracy and overall class-incremental accuracy across standard benchmarks (Split-CIFAR100, Split-ImageNet-R).

  2. 2

    Entropy-minimised ensemble gating: train lightweight input-conditional gating functions to softly aggregate task sub-network outputs during inference without requiring explicit discrete task labels, measuring classification accuracy and task-selection entropy.

  3. 3

    Reconstruction-based module selection: equip each task subnetwork with a compact autoencoding head or latent prior, evaluating input-level reconstruction loss at inference time to identify the source task before final prediction.

Have a different approach?

Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.

Free · 3 checks per day

Why it might fail

Inferring task identity from input features alone can be as difficult as solving the underlying multi-class classification problem, meaning task-routing errors directly bottleneck accuracy compared to replay-based class-incremental baselines.

Evidence

Each paper's own statement of the limitation, verbatim.

Nearest existing work

Related open problems

Continual Learning & Catastrophic Forgetting

Scope to testPartly addressed

A Standardized Transfer-Validation Benchmark for Methods Validated Only on Small-Scale Image Classification

Nearly every training-time method in the modern literature—continual learning, distillation, unlearning, robustness, federated aggregation—is validated exclusively on CIFAR/ImageNet-subset image classification with ResNet-18/50-class backbones, and each paper explicitly defers scale, architecture, and modality transfer to future work. Because every paper defers this simultaneously, the field has no evidence base at all for whether any of these methods survive the shift to transformers, higher resolutions, dense tasks, or non-vision data; practitioners choosing between methods are choosing based on measurements made in a regime none of them deploy in. The barrier is not one method's limitation but a collective, unpriced evaluation debt: nobody can tell which small-scale conclusions are regime-robust and which are artifacts of ResNet-18 on 32×32 images.

Possible candidate · 3/5 runs152 papers report this44% from 2025+

Continual Learning & Catastrophic Forgetting

Scope to testOpen

Characterizing and Reducing Pre-Trained ViT Dependence in Transfer-Based Methods

A wide range of methods built on frozen ImageNet-pretrained ViT features—whether for adapting to new tasks, constraining representations, or detecting anomalies—inherit an unexamined dependency: their guarantees hold only when the backbone's feature space is already good. Today nobody knows whether these methods degrade gracefully with weaker pretraining, generalize to domains far from ImageNet, or transfer to non-transformer architectures, because the dependence is never ablated. The consequence is that reported gains may be properties of the backbone, not of the methods themselves, and the methods are silently inapplicable wherever a strong ImageNet ViT does not exist.

Possible candidate · 3/5 runs20 papers report this78% from 2025+

Continual Learning & Catastrophic Forgetting

UnclassifiedOpen

Sub-Quadratic Covariance Sketching for Exemplar-Free Continual Learning

Exemplar-free continual learning methods increasingly rely on storing per-class feature statistics (prototypes and full covariance matrices) to generate pseudo-replay features or parameterize Mahalanobis classifiers without retaining raw inputs. However, storing a dense $D \times D$ covariance matrix for each of $C$ classes incurs $\mathcal{O}(C \cdot D^2)$ storage, which quickly exceeds device memory when scaling to large label spaces ($C \ge 1000$) or modern high-dimensional embeddings ($D \ge 1024$). Practitioners currently resort to naive diagonal approximations that discard inter-feature correlations and cause severe classification degradation under continual distribution shifts. As a result, covariance-based exemplar-free learning is structurally blocked from scaling to realistic large-vocabulary benchmarks on memory-constrained hardware.

Possible candidate · 3/5 runs6 papers report this50% from 2025+
Generated automatically, not curated by hand. Automated prior-work checks catch about a third of existing work, so treat this problem as a lead to investigate.