- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This manuscript studies cross-task generalization and forgetting in continual learning with pretrained models. It analyzes their relationship in a linear-model setting and investigates parameter updates in relation to pretrained weight magnitudes. Based on these analyses, it proposes DoubleFilter, which periodically filters updates that are disproportionately large or small relative to pretrained weights during sequential fine-tuning. The method optimizes the visual encoder without historical data replay, additional reference data, or task-specific modules. Experiments evaluate the method on class-incremental benchmarks and multi-domain task-incremental learning, with analyses of parameter subsets, masking strategies, and filtering intervals.
Critical strengths
- The manuscript addresses a practically relevant problem: preserving cross-task generalization while adapting a pretrained model continually, without additional reference data or task-specific architectural expansion.
- The evaluation includes multiple class-incremental configurations, parameter-subset analyses, alternative masking strategies, and filtering-interval studies. The manuscript also acknowledges its weaker performance on fine-grained classification.
Critical weaknesses
- Tables 2 and 3 contain no baselines newer than MagMax (2024), despite this being an ICLR 2027 submission. Relevant subsequent work, such as ConDU, MoE-Adapter and AFA, addresses continual adaptation and generalization of vision-language models on MTIL. Consequently, improvements over MagMax (2024) and proximity to ZSCL(2023) do not establish the method’s competitiveness against recent approaches.
- Filtering both large and small parameter updates already appears in Model Breadcrumbs. Related approaches such as NanoAdam and LiNeS also exploit parameter magnitudes or control update magnitudes to preserve pretrained capabilities. DoubleFilter’s potentially distinctive elements are its use of pretrained-relative magnitudes and periodic filtering during training. However, the manuscript does not adequately isolate their benefits from conventional magnitude-based filtering. A controlled comparison with absolute-magnitude two-sided filtering under the same training and filtering schedule is particularly important for establishing the contribution.
- The manuscript interprets small-magnitude pretrained weights as potentially undertrained or unimportant, and uses this interpretation to motivate suppressing their large downstream updates. However, parameter magnitude alone does not reliably establish training adequacy, functional importance, or transferable knowledge. The discussion in Appendix B does not sufficiently justify this interpretation. Moreover, the method measures relationships between final pretrained magnitudes and downstream updates, rather than observing consistency between pretraining and fine-tuning update trajectories. These interpretations should be qualified and supported by more direct empirical evidence.
- The reported comparisons change not only which parameters are updated, but also their layer distribution, effective update magnitude, and capacity to fit the current task. Although Table 4 attempts to match update volume, these factors remain insufficiently controlled. Consequently, the results do not establish whether the gains arise from identifying more generalizable updates or from a different regularization–plasticity trade-off. Comparisons at matched current-task performance, together with controls for update magnitude and layer allocation, would help distinguish these explanations.
Questions for authors
- In Table 4, DoubleFilter achieves higher Last accuracy, but Rand Drop and Smallest Update achieve both higher Transfer and lower reported forgetting. The explanation that these alternatives learn less is plausible, but requires per-task acquisition results or comparisons at matched plasticity. Furthermore, the comparison with ZSCL is competitive in Last accuracy only: DoubleFilter obtains 83.5 versus 83.6 in Last, but 63.3 versus 68.1 in Transfer. The manuscript should distinguish improved final task performance from superior generalization or retention, and calibrate its claims accordingly.
- The five improvements over MagMax on CIFAR100 and ImageNet-R range from only 0.08 to 0.35 percentage points, without reported uncertainty. The three-order averaging described for the earlier analyses is not clearly specified for the main experiments. DoubleFilter also underperforms MagMax and LwF across all reported CUB200 configurations. These results, together with the concentration of the main evaluation on CLIP ViT-B/16, limit the support for broad conclusions about generalizable parameter updates. Repeated main experiments and a targeted test of the proposed explanation for the fine-grained failure would strengthen the evidence.