acceptodds
Withdrawn from ICLR 2027

Rethinking Cross-Task Generalization in Continual Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual learning (CL) aims to sequentially learn tasks with reduced forgetting and improved cross-task generalization. Although the two goals are usually treated separately in previous works, our theoretical analysis shows that improving cross-task generalization helps reduce forgetting. Previous works address the cross-task generalization problem by introducing large-scale reference data or freezing the pre-trained weights. However, the former is computationally expensive, while the latter does not assess the model's generalization ability in full fine-tuning. To improve the cross-task generalization with full parameter tuning, we find that parameters with largest (weight) updates in fine-tuning can be vulnerable in cross-task generalization and suffer more from forgetting, while parameters that have consistent updates in pre-training and fine-tuning are more robust. Inspired by this, we propose a double filtering mask to filter weight updates in fine-tuning that are inconsistent with pre-trained weight magnitudes. Although simple to implement, our method significantly improves model's cross-task generalization and achieves comparable performance to CL models that require large-scale reference data.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Cross-Task Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pDD6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This manuscript studies cross-task generalization and forgetting in continual learning with pretrained models. It analyzes their relationship in a linear-model setting and investigates parameter updates in relation to pretrained weight magnitudes. Based on these analyses, it proposes DoubleFilter, which periodically filters updates that are disproportionately large or small relative to pretrained weights during sequential fine-tuning. The method optimizes the visual encoder without historical data replay, additional reference data, or task-specific modules. Experiments evaluate the method on class-incremental benchmarks and multi-domain task-incremental learning, with analyses of parameter subsets, masking strategies, and filtering intervals.

Critical strengths
  1. The manuscript addresses a practically relevant problem: preserving cross-task generalization while adapting a pretrained model continually, without additional reference data or task-specific architectural expansion.
  2. The evaluation includes multiple class-incremental configurations, parameter-subset analyses, alternative masking strategies, and filtering-interval studies. The manuscript also acknowledges its weaker performance on fine-grained classification.
Critical weaknesses
  1. Tables 2 and 3 contain no baselines newer than MagMax (2024), despite this being an ICLR 2027 submission. Relevant subsequent work, such as ConDU, MoE-Adapter and AFA, addresses continual adaptation and generalization of vision-language models on MTIL. Consequently, improvements over MagMax (2024) and proximity to ZSCL(2023) do not establish the method’s competitiveness against recent approaches.
  2. Filtering both large and small parameter updates already appears in Model Breadcrumbs. Related approaches such as NanoAdam and LiNeS also exploit parameter magnitudes or control update magnitudes to preserve pretrained capabilities. DoubleFilter’s potentially distinctive elements are its use of pretrained-relative magnitudes and periodic filtering during training. However, the manuscript does not adequately isolate their benefits from conventional magnitude-based filtering. A controlled comparison with absolute-magnitude two-sided filtering under the same training and filtering schedule is particularly important for establishing the contribution.
  3. The manuscript interprets small-magnitude pretrained weights as potentially undertrained or unimportant, and uses this interpretation to motivate suppressing their large downstream updates. However, parameter magnitude alone does not reliably establish training adequacy, functional importance, or transferable knowledge. The discussion in Appendix B does not sufficiently justify this interpretation. Moreover, the method measures relationships between final pretrained magnitudes and downstream updates, rather than observing consistency between pretraining and fine-tuning update trajectories. These interpretations should be qualified and supported by more direct empirical evidence.
  4. The reported comparisons change not only which parameters are updated, but also their layer distribution, effective update magnitude, and capacity to fit the current task. Although Table 4 attempts to match update volume, these factors remain insufficiently controlled. Consequently, the results do not establish whether the gains arise from identifying more generalizable updates or from a different regularization–plasticity trade-off. Comparisons at matched current-task performance, together with controls for update magnitude and layer allocation, would help distinguish these explanations.
Questions for authors
  1. In Table 4, DoubleFilter achieves higher Last accuracy, but Rand Drop and Smallest Update achieve both higher Transfer and lower reported forgetting. The explanation that these alternatives learn less is plausible, but requires per-task acquisition results or comparisons at matched plasticity. Furthermore, the comparison with ZSCL is competitive in Last accuracy only: DoubleFilter obtains 83.5 versus 83.6 in Last, but 63.3 versus 68.1 in Transfer. The manuscript should distinguish improved final task performance from superior generalization or retention, and calibrate its claims accordingly.
  2. The five improvements over MagMax on CIFAR100 and ImageNet-R range from only 0.08 to 0.35 percentage points, without reported uncertainty. The three-order averaging described for the earlier analyses is not clearly specified for the main experiments. DoubleFilter also underperforms MagMax and LwF across all reported CUB200 configurations. These results, together with the concentration of the main evaluation on CLIP ViT-B/16, limit the support for broad conclusions about generalizable parameter updates. Repeated main experiments and a targeted test of the proposed explanation for the fine-grained failure would strengthen the evidence.
Reviewer_uDgROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the relationship between cross-task generalization and forgetting in continual learning. The authors observe that large fine-tuning updates to parameters with small pre-trained magnitudes can harm cross-task generalization. Based on this observation, they propose DoubleFilter, which periodically filters weight updates based on their relative magnitudes to preserve pre-trained knowledge and mitigate forgetting.

Critical strengths
  1. The paper provides empirical evidence that suppressing large fine-tuning updates to small-magnitude pre-trained weights can improve generalization and reduce forgetting.
  2. The proposed DoubleFilter is a simple and intuitive approach that incorporates these empirical observations into a periodic weight-update filtering procedure without requiring additional data or parameters.
Critical weaknesses
  1. Unclear problem formulation and positioning. The paper would benefit from a clearer definition of "cross-task generalization" and its distinction from established concepts such as forward transfer, backward transfer, and OOD generalization. The connection between the theoretical definitions and experimental metrics could also be made more explicit to better motivate the baseline selection and evaluation. Additionally, a more thorough discussion of related works on selective parameter updating and pre-trained knowledge preservation would help clarify the novelty and advantages of the proposed method.
  2. Insufficient theoretical justification. Section 3 establishes upper bounds on generalization error and forgetting involving a shared optimal solution, but the implications for the relationship between generalization and forgetting could be clarified further. Table 1 could also better distinguish generalization from task-informed MTL initialization. Moreover, additional justification for the hypothesis that large updates to small pre-trained weights harm cross-task generalization would help strengthen the paper's central motivation.
Questions for authors

I would appreciate further clarification of the problem definition, positioning, and theoretical motivation to better understand and assess the technical contributions of the paper.

Reviewer_tfZdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how CLIP can mitigate forgetting of old tasks and maintain transferability to unseen tasks when adapting to new tasks in continual learning. The authors propose that the relationship between parameter update magnitudes and pretrained weight magnitudes may influence cross-task generalization, and introduce DoubleFilter to periodically filter parameter updates during full-parameter fine-tuning. Specifically, DoubleFilter removes large updates associated with small-magnitude pretrained weights and small updates associated with large-magnitude pretrained weights. By repeatedly applying these two filtering operations during training, the method aims to preserve cross-task generalization while maintaining the model's ability to learn new tasks.

Critical strengths
  1. The paper addresses a meaningful research problem. Continual learning is an important research direction, and improving cross-task generalization while mitigating catastrophic forgetting is valuable for developing models that can adapt to evolving tasks.

  2. The proposed DoubleFilter method is simple and easy to implement. Experimental results demonstrate modest performance improvements over some relatively outdated baseline methods.

Critical weaknesses
  1. The methodological novelty of this paper is limited, as the proposed DoubleFilter method largely builds upon existing ideas of magnitude-based parameter selection and update filtering. For example, [1] selects parameter updates based on their magnitudes for continual learning, while [2] investigates the relationship between pretrained weight magnitudes and fine-tuning updates. Compared with these studies, DoubleFilter mainly extends the filtering strategy to both large and small updates and applies it periodically during training. These modifications appear relatively straightforward, and the paper does not sufficiently clarify their distinct methodological contributions or demonstrate why they constitute a substantial advance over existing approaches. [1] MagMax: Leveraging Model Merging for Seamless Continual Learning. ECCV, 2024. [2] Pay Attention to Small Weights. NeurIPS, 2025.

  2. The experimental evaluation is insufficient, as most of the compared methods are relatively outdated. Although DoubleFilter achieves certain performance improvements over these baselines, the lack of comparisons with more recent and competitive continual learning methods makes it difficult to assess its effectiveness and competitiveness against current state-of-the-art approaches.

Questions for authors
  1. The proposed DoubleFilter method mainly extends existing magnitude-based parameter selection and update filtering strategies. Could the authors further clarify the specific methodological contributions of two-sided filtering and its periodic application, particularly in comparison with [1] and [2]? [1] MagMax: Leveraging Model Merging for Seamless Continual Learning. ECCV, 2024. [2] Pay Attention to Small Weights. NeurIPS, 2025.

  2. Section 3.2 provides upper bounds for cross-task generalization error and forgetting, both involving the distance between the initial weights and a shared optimum. However, these bounds do not directly establish that reducing generalization error leads to less forgetting. Could the authors provide further theoretical justification for this conclusion?

  3. Most of the compared methods are relatively outdated. Could the authors include more recent and competitive continual learning methods to better demonstrate the effectiveness of DoubleFilter?

  4. Table 4 shows that random dropping and selecting the smallest updates achieve higher Transfer scores but lower Last accuracy. Could the authors compare these methods under similar Last accuracy to better demonstrate the advantages of DoubleFilter?

  5. Table 4 shows that DoubleFilter outperforms Large Update Filter, but the two methods use different masking ratios and update volumes. Could the authors investigate whether the improvement persists under matched masking budgets?

  6. Since Table 5 includes filtering only at the end of each task, could the authors clarify whether the second filtering strategy remains beneficial under this setting?

  7. Could the authors report the actual training time to support the efficiency claims in Section 6.3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.