Continual Learning & Catastrophic Forgetting
Task-Agnostic Inference for Modular and Masked Continual Learning Architectures
Generated automatically from the limitations stated in 3 papers (CVPR, ICML, ICLR), listed under Evidence. It is not a paper, and it does not come from papers submitted to CSPaper.
The problem
Modular continual learning methods prevent catastrophic forgetting by allocating task-specific masks, sub-networks, or adaptive parameters during training. However, these methods structurally depend on having ground-truth task identifiers provided at test time to select the corresponding parameters. In practical deployments, incoming inputs arrive without task metadata, rendering task-dependent parameter masking unusable in class-incremental or task-agnostic settings. Without a reliable mechanism to resolve task identity or route inputs dynamically at test time, parameter-isolation architectures remain restricted to artificial task-incremental benchmarks.
Why it matters
Parameter-isolation and masked continual learning techniques become deployable in class-incremental and unlabelled streaming environments where test inputs lack task identifiers.
Ways to approach it
Prior-work checks are free with an account. Results someone already ran are shown to everyone.
- 1
Benchmark out-of-distribution task routing: implement task-specific density estimators (e.g., Mahalanobis distance or energy-based scores on intermediate representations) alongside frozen task modules, measuring routing accuracy and overall class-incremental accuracy across standard benchmarks (Split-CIFAR100, Split-ImageNet-R).
- 2
Entropy-minimised ensemble gating: train lightweight input-conditional gating functions to softly aggregate task sub-network outputs during inference without requiring explicit discrete task labels, measuring classification accuracy and task-selection entropy.
- 3
Reconstruction-based module selection: equip each task subnetwork with a compact autoencoding head or latent prior, evaluating input-level reconstruction loss at inference time to identify the source task before final prediction.
Have a different approach?
Describe how you would tackle this problem and we'll look for papers that already do it. Free; your text stays private.
Why it might fail
Inferring task identity from input features alone can be as difficult as solving the underlying multi-class classification problem, meaning task-routing errors directly bottleneck accuracy compared to replay-based class-incremental baselines.
Evidence
Each paper's own statement of the limitation, verbatim.
- Adaptive Plasticity Improvement for Continual LearningCVPR 2023
Requires access to task identity labels during both the training and inference stages.
- NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse NetworksICML 2022
Primary architecture relies on task labels being available during inference (task-incremental setting).
- Scalable and Order-robust Continual Learning with Additive Parameter DecompositionICLR 2020
Requires task identity at inference time to apply the correct task-specific masks and adaptive parameters.
Nearest existing work
- Overcoming Catastrophic Forgetting with Hard Attention to the TaskICML 2018
- Conditional Channel Gated Networks for Task-Aware Continual LearningCVPR 2020
- Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic ForgettingICML 2019
- Helpful or Harmful: Inter-Task Association in Continual LearningECCV 2022
- Semantic Drift Compensation for Class-Incremental LearningCVPR 2020
- Overcoming Catastrophic Forgetting by Bayesian Generative RegularizationICML 2021
- Layerwise Optimization by Gradient Decomposition for Continual LearningCVPR 2021
- A Combinatorial Perspective on Transfer LearningNeurIPS 2020
- Rehearsal-free Continual Language Learning via Efficient Parameter IsolationACL 2023
- Growing a Brain with Sparsity-Inducing Generation for Continual LearningICCV 2023
- Learning to Remember: A Synaptic Plasticity Driven Framework for Continual LearningCVPR 2019
- Continual Unsupervised Representation LearningNeurIPS 2019
- How catastrophic can catastrophic forgetting be in linear regression?COLT 2022
- Rehearsal-Free Modular and Compositional Continual Learning for Language ModelsNAACL 2024
- Representation Compensation Networks for Continual Semantic SegmentationCVPR 2022
Related open problems
Continual Learning & Catastrophic Forgetting
A Standardized Transfer-Validation Benchmark for Methods Validated Only on Small-Scale Image Classification
Nearly every training-time method in the modern literature—continual learning, distillation, unlearning, robustness, federated aggregation—is validated exclusively on CIFAR/ImageNet-subset image classification with ResNet-18/50-class backbones, and each paper explicitly defers scale, architecture, and modality transfer to future work. Because every paper defers this simultaneously, the field has no evidence base at all for whether any of these methods survive the shift to transformers, higher resolutions, dense tasks, or non-vision data; practitioners choosing between methods are choosing based on measurements made in a regime none of them deploy in. The barrier is not one method's limitation but a collective, unpriced evaluation debt: nobody can tell which small-scale conclusions are regime-robust and which are artifacts of ResNet-18 on 32×32 images.
Continual Learning & Catastrophic Forgetting
Characterizing and Reducing Pre-Trained ViT Dependence in Transfer-Based Methods
A wide range of methods built on frozen ImageNet-pretrained ViT features—whether for adapting to new tasks, constraining representations, or detecting anomalies—inherit an unexamined dependency: their guarantees hold only when the backbone's feature space is already good. Today nobody knows whether these methods degrade gracefully with weaker pretraining, generalize to domains far from ImageNet, or transfer to non-transformer architectures, because the dependence is never ablated. The consequence is that reported gains may be properties of the backbone, not of the methods themselves, and the methods are silently inapplicable wherever a strong ImageNet ViT does not exist.
Continual Learning & Catastrophic Forgetting
Sub-Quadratic Covariance Sketching for Exemplar-Free Continual Learning
Exemplar-free continual learning methods increasingly rely on storing per-class feature statistics (prototypes and full covariance matrices) to generate pseudo-replay features or parameterize Mahalanobis classifiers without retaining raw inputs. However, storing a dense $D \times D$ covariance matrix for each of $C$ classes incurs $\mathcal{O}(C \cdot D^2)$ storage, which quickly exceeds device memory when scaling to large label spaces ($C \ge 1000$) or modern high-dimensional embeddings ($D \ge 1024$). Practitioners currently resort to naive diagonal approximations that discard inter-feature correlations and cause severe classification degradation under continual distribution shifts. As a result, covariance-based exemplar-free learning is structurally blocked from scaling to realistic large-vocabulary benchmarks on memory-constrained hardware.